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ABSTRACT 

The  use  of  automation  in  uninhabited  aerial  vehicles  (UAVs)  is  being  explored  using 
simulation  to  develop  a  better  balance  of  system  autonomy  with  human  interaction  for 
optimal  human-system  performance.  Integrating  fixed  levels  of  automation  into 
elements  of  UAV  control  has  resulted  in  improved  performance  of  some  tasks,  but  less 
so  when  the  automation  does  not  fit  appropriately  to  changing  UAV  tasks  and 
environments.  As  a  consequence,  recent  research  has  focused  on  automation  that  can 
be  changed  during  UAV  operation.  The  aim  of  this  study  was  to  compare  the 
performance  of  operators  for  some  UAV  tasks  of  interest  under  automation  that  could 
be  changed  (adaptable),  fixed  automation,  and  no  automation.  A  secondary  aim  was  to 
examine  the  use  of  automation  and  its  effect  on  the  performance  of  these  tasks  in 
different  area  densities.  Thirty  eight  participants  from  the  Defence  Science  and 
Technology  (DST)  Group  Edinburgh  controlled  the  sensor  of  an  UAV  and  undertook 
typical  tasks  in  a  maritime  environment.  Sensor  control  was  manipulated  to  be  fully 
automated  (fixed),  completely  manual,  or  adjustable  by  the  participant  between 
automated  and  manual  modes  (adaptable).  The  results  showed  that  fixed  sensor 
control  produced  superior  performance  compared  to  manual  control  across  tasks,  but 
adaptable  control  produced  additional  time  improvements  compared  to  fixed  control. 
The  benefits  of  adaptable  automation  of  sensor  control  and  future  research  directions 
are  discussed. 
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Automation  of  Sensor  Control  in  Uninhabited  Aerial 

Vehicles 

Executive  Summary 

The  use  of  automation  in  uninhabited  aerial  vehicles  (UAVs)  is  currently  being 
explored  in  using  simulation  to  develop  a  better  balance  of  system  autonomy  with 
human  interaction  for  optimal  human-system  performance.  Integrating  fixed  levels  of 
automation  into  elements  of  UAV  control  has  shown  improved  performance  of  some 
tasks,  but  less  so  when  the  automation  does  not  fit  appropriately  to  changing  UAV 
tasks  and  environments.  As  a  consequence,  recent  research  has  focused  on  automation 
that  can  be  changed  during  UAV  operation,  otherwise  termed  adaptable  automation. 

This  report  describes  a  human-in-the-loop  simulation  study  conducted  to  extend  the 
knowledge  of  using  adaptable  automation  in  UAVs.  The  research  formed  the  basis  of  a 
master's  thesis  completed  through  the  University  of  Adelaide  under  a  Graduate 
Industry  Placement  arrangement  with  the  Defence  Science  and  Technology  (DST) 
Group.  Its  purpose  was  to  inform  UAV  use  by  the  Royal  Australian  Navy.  The 
objectives  of  the  study  were  to  compare  the  differences  in  performance  of  UAV  tasks 
under  automation  that  could  be  changed,  automation  that  was  fixed,  and  no 
automation  during  operation.  A  secondary  objective  was  to  examine  the  use  and  effect 
of  automation  on  performance  of  tasks  in  different  area  densities. 

Tasks  were  designed  to  be  representative  of  those  conducted  in  a  maritime  context, 
and  involved  reporting  details  about  a  specific  surface  vessel  in  a  particular  area  as 
well  as  reporting  on  the  total  number  of  surface  vessels  in  that  area.  Participants  were 
also  required  to  acknowledge  changes  in  system  status  gauges  when  they  occurred. 
The  automation  conditions  were  tested  at  the  camera  level  of  control,  otherwise 
referred  to  as  the  sensor,  and  as  such  all  flight  and  navigation  of  the  UAV  was 
predetermined  by  the  system.  Sensor  control  conditions  were  manipulated  such  that 
participants  performed  the  reporting  and  system  status  tasks  when  the  sensor  was 
controlled  manually,  when  it  was  fully  automated  and  could  not  be  controlled  by  the 
participant  (fixed),  and  when  participants  could  switch  between  full  automation  and 
manual  control  (adaptable).  The  area  density  was  also  manipulated  by  changing  the 
total  number  of  vessels  in  each  area. 

It  was  found  that  fully  automating  the  sensor  control  improved  reporting  performance 
compared  to  when  participants  had  to  operate  the  sensor  manually.  Additionally, 
when  participants  could  freely  switch  between  manual  and  fully  automated  sensor 
control,  improved  reporting  performance  was  also  found  compared  to  when  only 
manual  control  was  available.  When  the  automation  could  be  changed  during 
operation,  participants  produced  the  fastest  reports  overall  but  the  error  rate  remained 
no  different  to  that  when  the  sensor  was  fully  automated. 
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These  findings  suggest  that  adaptable  automation  at  the  sensor  control  level  should  be 
employed  for  superior  performance  of  operator  tasks  in  this  setting.  It  is  proposed  that 
expanding  the  number  of  operator  tasks  in  future  simulations  could  build  on  the 
benefits  of  adaptable  automation  use  identified  in  the  current  study,  as  well  as 
highlight  its  potential  impacts. 
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Uninhabited  Aerial  Vehicles  (UAVs)  have  become  indispensable  assets  in  defence 
organisations  around  the  world  allowing  reduced  radar  signatures,  increased  endurance, 
and  most  importantly  the  removal  of  humans  from  immediate  threat  (Hopcroft,  Burchat, 
&  Vince,  2006;  McCarley  &  Wickens,  2005).  With  exponential  increases  in  technology  over 
the  past  two  decades,  today's  UAVs  are  highly  automated  and  can  be  directed  to  follow  a 
pre-programmed  mission,  fly  to  designated  coordinates,  fly  specific  patterns,  correct  for 
course  deviations,  and  hold  above  a  particular  coordinate  or  target  (Arrabito  et  al.,  2010). 

It  is  argued  that  increased  UAV  automation  provides  benefits  such  as  simplified 
operations,  lower  operating  costs,  and  reduced  operator  workload  (Arrabito,  et  al.,  2010). 
However,  one  significant  consequence  of  automation  is  the  altered  role  of  the  human 
operator  from  manual  control  to  supervisory  or  managerial  control  of  the  UAV 
(Parasuraman,  Molloy,  Mouloua,  &  Hilburn,  1996).  As  a  result,  the  operator  may  have  to 
evaluate  or  finalise  automated  decisions,  diagnose  problems,  and  take  over  manually 
when  the  automation  fails  (Arrabito,  et  al.,  2010). 

With  the  advent  of  increased  levels  of  automation  (LOA)  within  UAV  systems,  there  has 
been  increasing  effort  to  design  systems  to  remove  the  need  for  multiple  operators  to 
control  one  vehicle  and  instead  have  one  operator  control  multiple  vehicles  (Cummings, 
Bruni,  Mercier,  &  Mitchell,  2007;  Dixon,  Wickens,  &  Chang,  2005;  Ruff,  Narayanan,  & 
Draper,  2002).  However,  single  operator  control  of  one  or  even  more  autonomous  vehicles 
needs  to  be  evaluated  to  address  the  balance  of  system  autonomy  with  human  interaction 
for  optimal  human-system  performance. 


1.1  Automation 

The  role  of  automation  in  systems  has  challenged  the  research  community  for  over 
50  years  with  a  range  of  pragmatic  and  philosophical  issues,  namely  the  failure  of 
automation  to  achieve  its  promised  benefits  and  how  automation  redefines  the  role  of 
humans  in  complex  systems  (Lee,  2008).  The  literature  on  human-automation  interaction  is 
extensive,  however  some  key  issues  have  emerged  that  have  been  termed  by  Parasuraman 
and  Riley  (1997)  as  human's  use,  misuse,  disuse,  and  abuse  of  automation.  These  issues  are 
still  relevant  over  10  years  later,  despite  major  technological  advances  during  this  period, 
and  will  be  discussed  (Lee,  2008). 

With  regard  to  human's  use  of  automation,  it  is  commonly  thought  that  the  user's 
engagement  in  automation  to  perform  functions  that  were  otherwise  performed  manually 
reduces  workload  and  mitigates  high  workload  situations.  On  the  other  hand,  it  has  been 
suggested  that  the  apparent  simplicity  of  automation  can  also  mask  its  actual  complexity, 
which  is  only  revealed  in  demanding  situations  (Woods,  1996;  Woods  &  Dekker,  2000). 

Problems  may  arise  when  humans  rely  on  poorly  performing  automation  through 
excessive  trust  in  the  automation,  engage  automation  in  situations  that  are  not  appropriate 
for  doing  so,  or  fall  prey  to  automation  biases  that  make  them  less  attentive  to 
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contradictory  information:  factors  that  all  contribute  to  humans'  misuse  of  automation.  Of 
particular  relevance  in  this  concern  is  the  monitoring  of  failures,  whereby  humans  tend  to 
neglect  automation  breakdowns.  Typically,  the  less  likely  a  failure  is  to  occur  the  more 
likely  the  human  operator  will  not  detect  an  automation  breakdown  when  one  actually 
occurs  (Lee,  2008). 

In  the  domain  of  disuse  of  automation,  humans  fail  to  engage  in  automation  where  it  could 
enhance  performance.  In  this  way,  humans  can  be  slow  to  accept  automation  as  it  may 
threaten  their  way  of  working,  they  may  have  not  developed  an  appropriate  level  of  trust 
in  its  capability,  or  the  automation  lacks  the  needed  functionality  (Lee,  2006).  Warnings 
also  play  a  role  in  this  area;  a  high  rate  of  false  alarms  in  systems  is  likely  to  undermine 
acceptance  and  lead  to  disuse  (Bliss,  2003). 

Abuse  of  automation  has  been  described  as  situations  in  which  automation  is  designed  and 
implemented  without  paying  sufficient  attention  to  its  effects  on  the  operators,  in  an  effort 
to  replace  unreliable  people  with  reliable  automation.  Here  exists  a  "replacement  fallacy" 
(Lee,  2008,  p.  407),  by  which  the  assumption  exists  that  failures  can  be  avoided  by 
automating  the  operator's  role,  however  this  ultimately  ignores  the  role  people  play  in 
accommodating  the  unexpected  and  preserving  an  efficient  system  (Woods  &  Dekker, 
2000). 


1.2  Conceptualising  UAV  automation 

A  practical  model  for  understanding  how  automation  plays  a  role  in  UAV  control  has 
been  provided  by  Cummings  et  al.  (2007)  as  shown  in  Figure  1.  Human  supervisory 
control  of  UAVs  is  hierarchical,  depicted  by  lower  and  higher  level  control  loops.  The 
innermost  and  therefore  the  lowest-order  loop  is  flight  control  of  the  UAV.  Within  this 
loop,  control  of  the  UAV  is  limited  to  basic  guidance  and  motion  control,  whereby  short¬ 
term  goals  such  as  keeping  the  aircraft  in  stable  flight  are  the  focus.  The  navigational  loop 
is  depicted  as  the  second-order  loop  in  the  hierarchy,  and  attends  to  mission  constraints 
such  as  routes  to  coordinates,  time  on  targets,  and  avoidance  of  threat  areas  and  no-fly 
zones.  The  highest-order  loop  is  represented  by  mission  and  payload  management 
whereby  incoming  information  from  sensors  must  be  monitored  in  order  to  make 
decisions  and  meet  overall  mission  requirements.  An  additional  system  health  and  status 
monitoring  loop  is  integrated  over  all  of  the  control  loops,  and  its  order  is  flexible  within 
the  hierarchy  as  depicted  by  its  dotted  line.  For  example,  should  system  health  decline  to  a 
critical  level,  tasks  within  this  loop  may  take  the  highest  priority,  however  when  system 
health  is  normal  monitoring  the  system  status  may  only  be  intermittent  and  secondary  to 
other  prioritised  tasks  within  the  hierarchical  control  loops.  The  authors  have  suggested 
that,  for  a  single  UAV  operator,  by  automating  tasks  required  in  lower  order  control  loops, 
task  demand  and  cognitive  reserve  can  be  freed  up  to  attend  to  tasks  in  higher  order 
control  loops. 
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System  Health  &  Status  Monitoring 


Mission  &  Payload  ~  ~ 
Management 


Sensor, 

communications,  & 
weapons  management 


~  ^  Navigation 


Planning  &  execution  for 
obstacle  avoidance  & 
route  headings 


~  {Auto)Pilot  "  ~  ~  ^  Flight  Controls 


--►i 


Pitch,  yaw,  airspeed  & 
altitude  control 


Figure  1:  Hierarchical  Control  Loops  for  a  Single  UAV  (Cummings,  et  al,  2007  pp.  4) 


1.3  Automation  in  UAVs 

Research  using  simulation  to  examine  automation  in  UAV  systems  has  suggested  that 
automating  some  tasks  required  of  UAV  operators  can  improve  performance  across  other 
tasks.  Wickens  and  Dixon  (2002)  had  participants  locate  surface  objects  in  the  sensor's  field 
of  view  (FOV)  while  an  UAV  was  in  flight.  In  addition,  Wickens  and  Dixon  (2002)  also 
required  participants  to  monitor  for  system  status  changes  depicted  by  gauge  needles  that 
would  move  from  a  green  to  a  red  zone  indicating  a  system  status  failure.  Participants  had 
to  press  a  "detect"  button  on  the  joystick,  then  use  a  keypad  to  input  which  system  had 
failed  and  a  required  coordinate.  If  the  system  status  failure  was  not  detected  within  30 
seconds,  then  the  gauge  needle  would  move  back  to  the  green  zone.  The  flight  control  loop 
of  the  UAV  was  completely  automated,  and  automation  was  experimentally  manipulated 
in  the  navigational  loop  of  the  UAV.  In  one  condition  participants  also  had  to  maintain  the 
UAV  on  a  straight  flight  path  to  a  particular  set  of  coordinates  by  manipulating  the 
joystick  left  and  right  along  its  x-axis,  while  in  another  condition  participants  previously 
inputted  "fly  to"  coordinates  that  had  the  UAV  fly  directly  to  those  coordinates  without 
requiring  maintenance  from  participants  to  stay  on  the  straight  flight  path.  Sensor 
manipulation  was  the  same  across  both  conditions  whereby  it  was  controlled  via  y-axis 
movement  of  a  joystick.  Participants  could  manipulate  the  sensor  to  view  straight  down  to 
the  land  surface  below  or  pan  up  to  45  degrees  toward  the  horizon. 

Compared  to  those  having  to  maintain  the  UAV  along  the  straight  flight  path,  those  that 
had  the  UAV  fly  the  path  automatically  were  subsequently  better  at  detecting  required 
surface  objects  with  the  sensor  (60%  versus  90%  detected  respectively).  This  suggested  that 
automating  the  navigational  requirement  of  operators  in  this  context  freed  up  task 
demand  and  subsequent  cognitive  reserve  to  allow  improved  surface  search  detection. 
Task  demand  was  reduced  for  those  with  automated  navigation  as  they  were  not  required 
to  manipulate  the  joystick  along  its  x  axis,  but  rather  only  focus  on  y  axis  movement  of  the 
sensor.  Additionally,  a  higher  proportion  of  system  status  failures  was  detected  and 
processed  for  those  with  automated  navigation,  although  system  status  processing  times 
between  those  with  automated  navigation  and  those  without  did  not  differ.  Overall, 
Wickens  and  Dixon  (2002)  provided  evidence  that  the  performance  of  operators,  in  tasks 
beyond  the  navigational  requirements  of  the  UAV,  would  likely  be  hindered  without 
automated  navigation  assistance.  These  findings  are  also  in  line  with  Cummings  et  al. 
(2007)  UAV  control  model,  where  higher  level  sensor  control  operation  and  mission 
requirements  were  degraded  by  lower  level  flight  and  navigation  requirements. 
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Building  on  the  study  by  Wickens  and  Dixon  (2002),  Dixon  et  al.  (2005)  assessed  operators 
in  exactly  the  same  experimental  conditions  as  in  the  former  study,  with  the  exception  of 
including  a  comparison  of  performance  measures  between  operators  of  two  UAVs  and 
operators  of  one  UAV.  Again  it  was  found  that  when  controlling  one  UAV,  detection  of 
required  surface  objects  was  superior  for  operators  with  automated  navigation  compared 
to  those  without  (92%  versus  79%  detected  respectively).  This  trend  also  remained  for  this 
task  when  controlling  two  UAVs  (76%  versus  52%  detected  respectively).  However,  there 
was  a  significant  drop  in  detection  of  required  surface  objects  overall  when  controlling 
two  UAVs  compared  to  one  UAV.  This  indicated  that  while  automating  navigation  in  this 
context  still  improved  performance  on  a  surface  search  task  compared  to  no  navigation 
automation,  this  task  was  negatively  impacted  by  the  introduction  of  two  UAVs  to  be 
controlled.  Under  dual  UAV  control,  workload  was  essentially  doubled  and  surface  search 
ability  deteriorated.  It  was  proposed  that  for  controlling  two  UAVs  or  potentially  more, 
automation  might  also  need  to  be  applied  at  the  highest-order  control  loop  of  sensor 
control  and  decision  making  (Cummings,  et  al.,  2007)  to  further  assist  in  reducing  operator 
workload  and  performance  deterioration. 

As  opposed  to  a  methodology  of  completely  automating  some  tasks  or  not  within  UAV 
systems,  other  research  has  examined  levels  of  automation  (LOAs)  for  particular  tasks 
along  a  continuum.  A  framework  has  been  outlined  from  no  automation  (the  human 
operator  makes  all  decisions  and  takes  all  actions),  through  management-by-consent 
(MBC;  the  system  suggests  a  course  of  action  or  small  set  of  alternatives  from  which  the 
human  accepts  or  rejects  or  chooses  a  preferred  course  of  action),  and  management-by- 
exception  (MBE;  the  human  is  informed  of  the  system's  intentions  and  given  the 
opportunity  to  veto  these  actions  within  a  set  time  frame),  to  complete  automation  (the 
system  makes  all  decisions  and  takes  all  actions)  (Parasuraman,  Sheridan,  &  Wickens, 
2000). 

Some  research  examining  this  approach  to  automation  in  UAVs  has  found  a  management- 
by-consent  LOA  to  be  most  effective  for  balancing  system  autonomy  with  human 
interaction  for  optimal  human-system  performance  (Liu,  Wasson,  &  Vincenzi,  2009;  Prinet, 
Terhune,  &  Sarter,  2012;  Ruff,  et  al.,  2002).  Ruff  et  al.  (2002)  had  participants  operate  UAVs 
to  seek  and  destroy  required  surface  targets.  Although  the  flight  control  loop  was 
completely  automated,  prompts  were  provided  to  participants.  A  prompt  was  provided 
when  the  fuel  of  the  UAV  had  reached  a  level  where  a  return  to  base  was  necessary,  the 
airspeed  had  approached  a  stall  speed,  or  when  the  altitude  had  approached  a  maximum 
or  minimum.  If  participants  accepted  the  prompt,  the  system  would  implement  the 
required  corrective  action  to  the  problem.  The  navigational  control  of  the  UAVs  was  done 
via  waypoints;  directional  drag-and-drop  lines  over  a  two  dimensional  map  of  an  area 
which  then  depicted  the  flight  path  of  the  UAV.  All  participants  manipulated  waypoints 
so  that  the  UAVs  flew  to  the  surface  targets  which  were  also  depicted  on  the  two 
dimensional  map  of  the  area.  The  authors  did  not  mention  any  human-system  interaction 
with  regard  to  the  sensor  control,  so  it  was  assumed  this  process  was  actioned  by  the 
system.  Users  could  traverse  an  additional  three-dimensional  map  of  the  area,  much  like 
an  invisible  spectator,  in  order  to  view  the  environment  of  the  area,  but  this  was  artificially 
created  for  the  simulation  and  not  a  view  from  the  sensor  on  an  UAV  itself.  Despite  this 
lack  of  detail,  the  authors  included  a  prompt  in  the  mission  and  payload  management 
control  loop  whereby  the  system  notified  the  operator  when  a  surface  target  was  detected 
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by  an  UAV.  If  participants  accepted  the  prompt,  the  system  would  shoot  the  target  when  it 
was  within  shooting  range. 

Ruff  et  al.  (2002)  implemented  three  conditions  in  their  study.  A  MBC  condition  involved 
the  system  offering  participants  all  prompts  as  described,  but  no  action  would  be  taken 
unless  participants  had  accepted  the  prompt.  On  the  other  hand,  a  MBE  condition 
involved  all  the  same  prompts  existing  as  in  the  MBC  condition,  but  the  system 
automatically  acted  after  three  seconds  unless  participants  rejected  the  prompt  prior  to  the 
action.  The  authors  also  referred  to  a  baseline  condition,  termed  as  a  manual  condition,  but 
did  not  explicitly  describe  what  this  was.  No  prompting  existed  in  this  condition,  and  it 
was  assumed  that  operators  were  then  required  to  monitor  gauges  of  flight  control  and 
manipulate  waypoints  accordingly,  as  well  as  manually  implement  the  shooting  of  targets. 
Nonetheless,  Ruff  et  al.  (2002)  reported  greater  mission  efficiency  from  those  in  the  MBC 
condition  compared  to  those  in  the  manual  and  MBE  conditions,  although  caution  should 
be  exercised  when  interpreting  this  finding.  Mission  efficiency  was  defined  as  the  total 
number  of  surface  targets  destroyed  divided  by  the  total  number  of  missiles  fired,  yet  it 
was  not  made  explicit  by  the  authors  how  operators  might  have  fired  more  missiles  than 
surface  targets,  and  specifically  how  this  performance  measurement  could  be  affected  by  a 
LOA  being  applied.  Ruff  et  al.  (2002)  also  reported  a  reduced  number  of  ''hit  points"  (Ruff, 
et  al.,  2002  pp.  343)  for  those  in  the  MBC  condition  compared  to  those  in  the  manual  and 
MBE  conditions.  If  an  UAV  was  navigated  to  within  "100  units"  (distance  of  a  unit  was 
undefined;  Ruff,  et  al.,  2002  pp.  343)  of  a  surface  target,  this  was  defined  as  a  violation,  and 
one  hit  point  was  received  for  every  250  milliseconds  of  violation.  The  findings  suggested 
that  applying  more  automation,  or  moving  to  a  higher  LOA  such  as  MBE,  in  a  human- 
system  UAV  design  may  not  always  improve  overall  human-system  performance. 
Additionally,  Ruff  et  al.  (2002)  pointed  out  that  their  participants  actually  preferred  the 
MBC  condition  compared  to  the  two  other  conditions,  and  suggested  that  this  LOA 
assisted  participants  in  attending  to  tasks  but  effectively  maintained  their  awareness  of  the 
system's  events  and  activities  being  performed. 

Other  research  (Liu  et  al.,  2009)  has  supported  findings  from  Ruff  et  al.  (2002)  that  MBC 
might  be  a  more  appropriate  approach  compared  to  MBE  for  optimal  human-system 
performance  in  UAVs.  Liu  et  al.  (2009)  had  participants  monitor  incoming  images  taken 
from  the  sensors  of  UAVs.  In  this  simulation  research,  many  of  the  UAV  functions  were 
already  completely  automated.  All  flight,  navigation,  and  sensor  controls  were  actioned  by 
the  system.  An  "Automatic  Target  Recognizer  (ATR)"  (Liu,  et  al.,  2009  pp.  799)  in  the 
system  placed  a  red  box  around  suspected  targets  in  the  images  taken  from  the  UAV 
sensor.  The  ATR  was  80  per  cent  reliable,  thus  if  its  actions  were  incorrect  the  participant 
had  to  add  or  delete  red  boxes  on  the  images  themselves.  In  a  MBC  condition,  participants 
were  prompted  to  accept  or  reject  images.  If  accepted,  the  image  was  removed  and  the 
next  image  from  a  queue  of  images  that  had  been  taken  by  the  sensor  would  appear.  The 
authors  were  unclear  in  describing  the  reject  function,  but  presumably  rejecting  the  image 
would  allow  manipulation  of  the  red  boxes,  after  which  the  image  could  be  accepted. 
Participants  in  the  MBC  condition  were  compared  to  those  in  a  MBE  condition,  whereby 
the  system  automatically  accepted  the  image  after  15  seconds  unless  the  participant 
intervened  earlier  or  pressed  a  hold  button  to  allow  more  time.  Liu  et  al.  (2009)  found  that 
those  in  the  MBC  condition  yielded  shorter  image  processing  times  than  those  in  the  MBE 
condition,  although  there  were  no  differences  in  error  rates  between  conditions. 
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Further  to  Ruff  et  al.  (2002)  and  Liu  et  al.  (2009),  recent  research  by  Prinet  et  al.  (2012)  had 
participants  undertake  a  surface  target  detection  and  a  route-replanning  task.  In  the 
surface  detection  task,  participants  had  to  monitor  video  feeds  from  UAVs  as  they  were 
approaching  areas  where  a  target  might  have  been  present,  and  responded  by  pressing  a 
corresponding  button  on  the  system  if  they  detected  a  target,  were  unsure  if  there  was  a 
target,  or  were  confident  that  no  target  was  present.  In  the  route-replanning  task, 
circumstances  would  change  in  the  environment,  which  included  the 
addition/ removal/ moving  of  a  target,  activation  of  no-fly  zones,  poor  weather  conditions, 
or  a  fuel  leak  on  an  UAV,  which  required  participants  to  replan  a  route  for  an  UAV.  The 
lowest  order  flight  control  and  highest  order  sensor  control  loops  were  actioned  by  the 
system,  while  navigational  control  was  actioned  by  drag  and  drop  waypoints.  In  one  MBC 
condition,  labelled  ''intermediate"  (Prinet,  et  al.,  2012  pp.  424),  participants  could  replan  an 
UAV's  waypoints  by  choosing  one  of  three  different  replanning  options  generated  by  the 
system  in  response  to  the  environmental  change.  Alongside  each  of  these  options  were 
three  10-point  scores  with  10  being  the  best  possible  score.  The  first  score  represented  the 
time  for  the  UAV  to  reach  base,  the  second  represented  the  risk  of  the  UAV  being  shot 
down,  and  the  third  represented  the  percentage  of  targets  visited  at  the  time  the 
replanning  options  were  generated.  In  another  MBC  condition,  labelled  "automated" 
(Prinet,  et  al.,  2012  pp.  424)  participants  could  replan  an  UAV's  waypoints  by  choosing  one 
different  replanning  option  with  its  three  10-point  scores  generated  by  the  system.  For 
both  MBC  conditions,  participants  could  accept  a  new  proposed  set  of  waypoints  for  an 
UAV,  or  the  original  flight  path  would  remain  active  if  the  participant  rejected  the 
proposed  flight  path.  In  addition  to  the  two  MBC  conditions,  a  "manual"  condition  was 
included  (Prinet,  et  al.,  2012  pp.  424)  whereby  the  system  did  not  generate  replanning 
options  and  participants  had  to  drag  and  drop  new  waypoints  to  address  the  environment 
change  that  required  replanning. 

Prinet  et  al.'s  results  (2012)  followed  trends  reported  in  previous  research  (Liu,  et  al.,  2009; 
Ruff,  et  al.,  2002),  finding  that  participants  in  the  two  MBC  conditions  correctly  identified 
more  surface  targets  and  had  shorter  replanning  times  than  those  in  the  manual  condition. 
On  the  other  hand,  the  authors  also  found  some  contradictory  results,  whereby  those  in 
the  MBC  conditions  had  the  largest  proportion  of  UAVs  shot  down  as  a  result  of  being 
present  in  no-fly  zones  compared  to  those  in  the  manual  condition.  Here  the  authors 
pointed  out  a  common  problem  with  not  being  able  to  change  a  LOA  when  attending  to  a 
particular  UAV  task.  In  their  study,  no-fly  zones  were  updated  through  the  use  of  a  live 
chat  box  from  mission  command.  Known  no-fly  zones  were  inputted  into  the  system  prior 
to  the  mission  commencing,  thus  the  system  could  calculate  routes  for  the  UAVs 
accordingly,  but  changes  from  live  updates  were  not  able  to  be  programmed  into  the 
system.  As  a  result,  participants  in  the  MBC  conditions  with  only  the  option  to  continue  an 
UAV  along  its  original  flight  path  or  accept  a  new  one  generated  by  the  system  were  at 
greater  risk  of  directing  the  UAV  into  a  no-fly  zone  compared  to  those  who  could  freely 
manipulate  flight  paths  at  any  time  in  the  manual  condition. 

One  of  the  challenges  in  determining  the  appropriate  balance  of  system  autonomy  with 
human  interaction  for  optimal  human-system  performance  is  that  different  tasks  have 
been  automated  in  different  studies,  with  little  overlap  between  studies  to  enable  direct 
comparison.  Tasks  have  varied  from  surface  object  recognition  (Dixon,  et  al.,  2005; 
Wickens  &  Dixon,  2002),  to  seek  and  destroy  operations  (Ruff,  et  al.,  2002),  image  analysis 
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(Liu,  et  al.,  2009),  and  target  detection  (Prinet,  et  al.,  2012),  and  complexity  has  been  further 
introduced  by  applying  different  LOAs  on  these  tasks.  Nonetheless,  several  trends  are 
apparent.  Firstly,  following  Gumming  et  al.'s  (2007)  hierarchical  control  loop,  complete 
automation  of  the  flight  control  loop  and  some  automation  of  the  navigation  control  loop 
may  serve  for  better  human-system  performance  of  tasks  in  the  mission  and  payload  loop 
(Dixon,  et  al.,  2005;  Wickens  &  Dixon,  2002).  Secondly,  applying  automation  in  the 
navigation  and  mission  and  payload  loops  may  assist  human-system  performance  on 
particular  tasks,  but  the  LOA  applied  may  impede  this  performance  if  the  system  acts 
without  human  involvement  (Liu,  et  al.,  2009;  Ruff,  et  al.,  2002)  or  is  not  responsive  to 
changes  in  the  environment  (Prinet,  et  al.,  2012). 


1.4  Adaptable  automation 

A  criticism  of  applying  a  LOA  to  assist  human-system  performance  has  been  that  once  a 
LOA  is  designed  into  a  system  it  cannot  be  changed  during  operation,  and  thus  it  is  fixed 
(Miller,  Funk,  Goldman,  Meisner,  &  Wu,  2005;  Miller  &  Parasuraman,  2007).  This  is  of 
particular  concern  for  the  operation  of  UAVs,  whereby  changing  environments  and  tasks 
may  demand  more  or  less  interaction  from  the  user  with  elements  of  the  system  at  varying 
times,  but  the  applied  LOA  does  not  correspond  with  these  changes.  To  accommodate 
changing  environments  and  tasks  of  UAV  systems,  flexible  automation  has  been 
investigated,  which  allows  LOAs  to  change  based  on  user  need  and  workload  demand. 
Two  types  of  flexible  automation  within  UAV  systems  have  been  researched;  those  that 
are  system  driven  (adaptive)  and  those  that  are  user  driven  (adaptable)  (Miller,  et  al., 
2005).  Adaptive  automation,  based  on  pre-defined  rules  built  into  the  system,  shifts 
elements  of  the  system  into  a  lower  or  higher  LOA  depending  on  operator  interaction  and 
frequency  of  inputs.  Conversely,  in  adaptable  automation  the  user  shifts  elements  of  the 
system  into  a  lower  or  higher  LOA  when  they  deem  appropriate.  Miller,  Pelican  and 
Goldman  (2000)  compared  adaptive  and  adaptable  automation  and  concluded  that 
adaptive  automation  may  have  the  advantage  of  reducing  workload  when  needed,  but 
may  also  result  in  increased  unpredictability  in  the  system  given  the  user  has  no  control 
over  the  changes  in  LOA  that  may  occur  (Billings  &  Woods,  1994;  Barter,  Woods,  & 
Billings,  1997).  Adaptable  automation,  on  the  other  hand,  may  have  the  advantage  of 
reducing  unpredictability  of  the  system  as  the  user  can  knowledgably  choose  a  LOA  when 
required.  However,  adding  the  task  of  setting  a  LOA  to  the  user's  list  may  also 
inadvertently  increase  his/her  workload  (Bailey,  Scerbo,  Freeman,  Mikulka,  &  Scott,  2006). 

Adaptive  automation  has  led  to  improved  performance  on  some  UAV  tasks  over  no 
automation  (e.g.  Hou,  Kobierski,  &  Brown,  2007;  Parasuraman,  Cosenzo,  &  De  Visser, 
2009)  and  fixed  automation  (e.g.  Calhoun,  Ruff,  Spriggs,  &  Murray,  2012;  Parasuraman,  et 
al.,  2009),  but  very  little  empirical  research  has  examined  adaptable  automation  in  an  UAV 
context,  particularly  in  comparison  to  fixed  automation.  One  study  has  recently  suggested 
that  adaptable  automation  may  be  more  beneficial  than  adaptive  automation.  Kidwell, 
Calhoun,  Ruff,  and  Parasuraman  (2012)  had  participants  monitor  a  two  dimensional  map 
that  depicted  UAVs  flight  paths,  and  required  them  to  detect  and  acknowledge  the 
appearance  of  a  map  symbol  representing  a  hostile  aircraft  if  one  appeared  at  any  time. 
The  authors  also  tasked  participants  with  counting  the  number  of  specified  targets  in 
images  taken  from  the  sensors  of  the  UAVs.  As  in  Liu  et  al.'s  (2009)  study,  images  were 
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taken  by  UAVs  and  a  90%  reliable  ATR  was  used  to  estimate  the  number  of  specified 
targets  in  each  image.  Kidwell  et  al  (2012)  had  all  flight  control,  navigation,  and  sensor 
control  actioned  by  the  system.  In  an  adaptable  condition,  the  LOA  of  the  ATR  started  in  a 
low  automated  setting,  whereby  the  number  of  specified  targets  in  the  image  had  to  be 
inputted  by  the  participant.  If  the  ATR  was  moved  to  a  medium  setting,  the  system 
provided  a  recommendation  as  to  the  number  of  specified  targets  in  the  image.  If  the  ATR 
was  moved  to  a  high  setting,  the  system  also  provided  a  recommendation  as  to  the 
number  of  specified  targets  in  the  image,  but  inputted  this  number  for  the  participant  after 
20  seconds  unless  the  participant  vetoed  the  action  prior  to  this  time.  In  the  adaptive 
condition,  the  LOA  of  the  ATR  also  started  in  the  low  setting,  but  changed  according  to 
pre-built  rules  on  processing  times  for  the  required  tasks. 

In  their  research,  Kidwell  et  al.  (2012)  noted  that  for  the  highest  prioritised  task,  detecting 
and  acknowledging  a  hostile  aircraft,  detection  rate  was  significantly  higher  for 
participants  in  the  adaptable  compared  to  the  adaptive  condition,  suggesting  that  the  act 
of  delegating  the  LOA  may  have  served  to  better  keep  the  operator  aware  of  and  alert  to 
unexpected  stimuli.  Interestingly,  the  authors  reported  a  greater,  although  not  statistically 
significantly  so,  mean  response  time  in  the  detection  and  acknowledgement  task  in  the 
adaptable  compared  to  the  adaptive  condition.  Greater  cognitive  resources  may  have  been 
demanded  by  those  participants  who  had  to  delegate  a  LOA,  resulting  in  slightly 
increased  processing  time.  However,  it  appeared  that  a  loss  of  speed  was  traded  for 
improved  accuracy  in  this  condition.  For  the  image  analysis  task  itself  it  does  not  appear 
that  a  speed  accuracy  trade-off  existed.  The  authors  reported  a  slightly  slower,  although 
non-significantly  so,  task  completion  time  in  the  adaptable  compared  to  the  adaptive 
condition,  but  failed  to  find  any  difference  between  conditions  with  regard  to  image 
analysis  accuracy. 

Other  research  has  also  looked  at  the  human  delegating  LOAs  compared  to  no 
automation.  Shaw  et  al.  (2010)  had  operators  control  three  UAVs  to  detect  and  track 
specific  surface  vehicles  within  three  designated  areas.  If  a  specific  surface  vehicle 
travelled  to  an  armoury  location  to  become  weaponised,  operators  had  to  prosecute  the 
surface  vehicle  via  control  of  the  UAVs.  An  additional  task  required  by  operators  was  to 
identify  and  mark  or  ''paint"  (Shaw,  et  al.,  2010  pp.  1499)  civilian  vehicles  in  the 
designated  areas.  In  their  study,  all  flight  control  of  the  UAVs  was  automated,  navigation 
was  controlled  via  manipulation  of  waypoints,  and  sensor  control  was  managed  by  the 
operator  via  joystick  manoeuvring.  The  authors  compared  the  performance  of  operators 
under  manual  control  to  that  of  operators  who  were  able  to  implement  a  switch  to  a  LOA. 
Under  manual  control,  operators  had  to  drag  and  drop  waypoints  to  implement  flight 
paths  for  the  UAVs,  and  had  to  manipulate  the  sensor  in  order  to  detect  and  track  specific 
surface  vehicles  in  the  designated  areas.  Additionally,  those  in  the  manual  condition  had 
to  manually  task  prosecution  of  the  specific  surface  vehicle  if  it  became  weaponised.  In  a 
second  condition,  operators  could  switch  between  manual  and  a  high  LOA.  The  high  LOA 
permitted  three  "plays"  (Shaw,  et  al.,  2010  pp.  1450),  which  included  setting  all  UAVs  on 
automatic  flight  paths  with  the  sensors  slewed  to  continually  hold  the  entire  designated 
areas  in  their  FOVs,  setting  the  UAVs  to  automatically  prosecute  a  weaponised  surface 
vehicle,  or  setting  the  first  option  with  the  exception  of  manoeuvring  the  sensor  of  one 
UAV.  In  a  third  condition,  manual  and  a  high  LOA  were  available  for  use,  as  well  as  an 
intermediate  LOA.  The  intermediate  LOA  allowed  participants  to  scroll  through  one  UAV 
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at  a  time  and  set  an  automatic  flight  path  with  a  slewed  sensor  on  the  designated  area. 
These  three  conditions  were  assessed  across  two  workload  conditions  changing  the 
number  of  civilian  vehicles  presented.  In  a  low  workload  condition,  19  civilian  vehicles 
were  presented,  while  in  a  high  workload  condition  64  civilian  vehicles  were  presented. 

Shaw  et  al.  (2010)  found  that  tracking  accuracy,  measured  as  the  number  of  vehicles  that 
were  actually  tracked  against  the  number  that  could  potentially  be  tracked,  was 
significantly  better  in  the  third  condition,  where  all  LOAs  were  available  for  the  operator 
to  shift  between,  compared  to  the  first  or  second  condition  (61.7%  versus  46.1%  and  49.4% 
respectively).  Although  the  authors  did  not  report  which  LOA  the  simulation  started  in  or 
whether  shifts  were  actually  made  by  operators  who  had  the  option  of  using  a  LOA,  these 
findings  suggested  that  operators  with  three  LOAs  available  for  use  were  better  able  to 
perform  a  required  task  than  were  those  who  had  no  automated  assistance  or  only  one 
LOA  to  shift  to. 

Additionally,  Shaw  et  al.  (2010)  also  found  that  participants  with  all  LOAs  available  for 
use  were  faster  in  dealing  with  an  unexpected  event  compared  to  those  with  only  one 
LOA  to  use  or  no  automated  assistance.  The  unexpected  event  was  a  pop-up  target  which 
required  detection  and  prosecution,  but  appeared  outside  of  the  designated  areas.  For 
operators  to  attend  to  this  pop-up  target,  manual  control  had  to  be  used  to  drag  and  drop 
waypoints  in  order  to  navigate  an  UAV  toward  it,  as  the  LOAs  available  in  the  system 
only  applied  to  tasks  being  performed  in  the  designated  areas.  The  findings  demonstrated 
the  benefit  of  adaptable  automation  in  an  UAV  context,  whereby  automation  was 
delegated  accordingly  by  the  operator  not  only  to  better  perform  a  tracking  task,  but  also 
to  deal  with  unexpected  changes  in  the  environment. 

Building  on  the  study  by  Shaw  et  al.  (2010),  Miller  et  al.  (2011)  replicated  the  study  and 
found  similar  results  with  regard  to  tracking  accuracy  and  time  taken  to  deal  with  an 
unexpected  event.  Both  studies  also  found  that  the  manipulation  of  workload  affected 
tracking  accuracy  and  time  to  respond  to  the  unexpected  event,  whereby  low  workload 
produced  better  tracking  accuracy  and  faster  response  time  to  the  unexpected  event  than 
did  high  workload.  With  the  common  assumption  that  automation  may  help  mitigate  high 
workload  (Lee,  2008)  it  would  have  been  interesting  if  both  sets  of  authors  additionally 
assessed  when  particular  LOAs  were  shifted  to  and  if  participants  used  the  high  LOA 
more  when  it  was  available  for  use  in  the  high  workload  condition. 

In  summary,  research  has  suggested  that  automation  use  in  UAVs  offers  benefits  for  some 
tasks,  particularly  if  the  lowest  order  of  flight  control  is  completely  automated  by  the 
system.  Applying  LOAs  at  higher  orders  of  control  can  also  improve  human-system 
performance,  but  this  may  be  dependent  on  what  tasks  are  required  and  when.  Changing 
tasks  and  environments  have  focused  research  more  recently  on  flexible  automation  use  in 
UAVs,  and  while  system  driven  adaptive  automation  has  been  explored,  user  driven 
adaptable  automation  requires  further  investigation. 
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1.5  The  current  study 

While  adaptive  automation  has  been  shown  to  improve  the  performance  of  some  UAV 
tasks  when  compared  to  unchanging  LOAs  (Calhoun  et  al.,  2012;  Parasuraman  et  al., 
2009),  there  is  a  need  for  research  comparing  the  performance  of  UAV  tasks  under 
adaptable  automation  and  unchanging  LOAs.  Furthermore,  comparisons  between 
empirical  studies  are  made  inherently  difficult  given  the  extensive  range  of  tasks  that 
UAVs  and  their  operators  can  perform.  The  aim  of  this  study  was  to  compare  the 
performance  of  operators  for  some  UAV  tasks  of  interest  under  adaptable  automation, 
fixed  (unchanging)  automation,  and  no  automation.  A  secondary  aim  was  to  examine  the 
use  of  automation  and  its  effect  on  the  performance  of  these  tasks  in  different  area 
densities.  Based  on  the  available  previous  research  (Dixon  et  al.,  2005;  Kidwell  et  al.,  2012; 
Liu  et  al.,  2009;  Miller  et  al.,  2011;  Prinet  et  al.,  2012;  Ruff  et  al.,  2002;  Shaw  et  al.,  2010; 
Wickens  &  Dixon,  2002)  several  hypotheses  were  generated: 

1.  Fixed  automation  will  improve  performance  on  tasks  compared  to  no  automation. 

2.  Adaptable  automation  will  improve  performance  on  tasks  compared  to  no 
automation. 

3.  An  accuracy  over  speed  trade  off  will  exist  under  adaptable  compared  to  fixed 
automation;  accuracy  will  improve  at  the  expense  of  longer  response  times  in 
adaptable  compared  to  fixed  automation  (exploratory). 

4.  Users  will  rely  more  on  fixed  automation  when  under  higher  workload 
(exploratory). 
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2.  Method 


2.1  Participants 

Forty  staff  and  students  from  the  Defence  Science  and  Technology  (DST)  Group 
Edinburgh  volunteered  to  participate  in  the  study  following  an  intranet  daily  news 
advertisement.  The  advertisement  invited  those  interested  to  take  part  in  UAV  simulation 
research,  and  stated  that  no  experience  was  required  as  all  training  would  be  provided. 
Those  that  wished  to  participate  were  required  to  have  normal  vision  and  colour 
recognition.  One  individual  did  not  meet  the  inclusion  criteria  and  another  did  not 
complete  the  study  due  to  a  technical  failure  in  the  simulation.  The  final  data  set 
comprised  38  participants  (6  females  and  32  males)  with  ages  ranging  from  24  to  53  years 
(M  =  36.1,  SD  =  8.05). 


2.2  Measures/Apparatus 

2.2.1  Demographics 

Data  were  collected  on  age,  gender  and  gaming  experience  over  the  past  four  weeks  using 
a  5-point  Likert  scale  (1  =  "Not  at  all",  2  =  "Less  than  1  hour  per  week",  3  =  "1  to  3  hours 
per  week",  4  =  "4  to  6  hours  per  week",  5  =  "7  or  more  hours  per  week").  The  highest  level 
on  this  scale  typifies  an  experienced  gamer  (Strobach,  Frensch,  &  Schubert,  2012). 

2.2.2  Visual  screening  assessments 

Visual  acuity  was  measured  using  the  Hanks  Near  Point  Eye  Chart  (Hanks  &  Hanks, 
2005).  Normal  vision  is  indicated  by  the  participant  reading  a  printed  text  in  5-point  font 
size  at  a  distance  of  40  cm  without  errors. 

Colour  blindness  was  assessed  using  the  Ishihara  Colour  Blind  Test  (Ishihara,  1917)  which 
measures  colour  vision  deficiency  of  congenital  origin.  Twenty  four  plates  each  contain  a 
circle  of  dots  appearing  randomised  in  colour  and  size.  The  numerals  seen  on  plates  1-17 
are  stated.  For  those  that  are  unable  to  read  numerals,  plates  18-24  are  used  and  the 
winding  lines  between  the  two  "X"s  are  traced.  For  plates  1-15,  if  13  plates  are  read 
normally  this  indicates  normal  colour  vision.  Nine  or  less  plates  read  normally  indicates 
colour  vision  deficiency. 

2.2.3  Equipment  -  software  and  hardware 

Virtual  Battlespace  2  (VBS2)  v2.02  software  developed  by  Bohemia  Interactive  was  used  to 
simulate  the  UAV  and  the  environment  in  which  it  operated.  The  software  was  run  on  an 
Intel  17  Workstation  (Windows  7  64-bit  operating  system,  8  gigabytes  of  RAM,  Nvidia 
GTX690  graphics  card,  processor  dock  speed  of  3.50  GHz),  and  dual  monitors  (Dell 
30  inch  Ultrasharp  3011,  resolution  2560  x  1600,  60  Hz  and  HP  20  inch  LP2065,  resolution 
1600  X  1200,  60  Hz).  Accessories  included  a  HP  mouse  (Model  M-UAE96),  HP  keyboard 
(Model  KU-0316),  and  Microsoft  Sidewinder  Precision  2  joystick.  To  model  the  required 
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UAV  functionality,  both  internal  VBS2  scripts  and  external  scripts  written  in  Groovy  were 
developed. 

2.2.4  UAV  simulation 

All  flight  and  navigation  control  of  the  UAV  was  automated.  For  the  purpose  of 
experimental  rigour,  take-off  and  landing  were  removed  and  the  participants  were  only 
exposed  to  the  UAV  in  flight.  Additionally,  the  flight  was  always  set  at  1500  feet  in 
altitude  with  a  speed  of  100  kilometres  per  hour.  The  UAV  always  flew  in  a  circular  loiter 
with  a  loiter  radius  of  700  metres. 


Participants  controlled  the  camera  attached  to  the  UAV,  referred  to  as  the  sensor,  via 
visual  feedback  on  the  main  screen  (Dell  30  inch  Ultrasharp  3011)  combined  with  the 
movement  of  the  joystick.  Designated  buttons  on  the  joystick  controlled  locking  (holding 
the  sensor  at  a  point  irrespective  of  UAV  movement),  and  zooming  in  and  out  functions. 
Figure  2  shows  the  main  screen  and  all  its  elements. 


Mini  map 
UAV 

flight  path 

VOI 
location 

VOI 

lat/long 

coordinates 

Sensor 

crosshair 

lat/long 

coordinates 


Sensor  direction  UAV  direction  UAV  speed 


UAV  altitude 


Visible  when 
sensor  lock 
function  engaged 


_  Sensor 
crosshair 


Current  zoom 


Sensor  mode 


status 

indicators 


Time  remainina  in  area 


Figure  2:  Visual  from  main  screen  in  the  study  simulation. 


An  UAV  model  was  created  by  using  a  VBS2  Scan  Eagle  model  for  the  flight  dynamics  and 
implementing  a  custom  sensor  view.  The  custom  sensor  view  was  created  to  provide 
required  functionality,  experimental  control  and  data  capture.  The  sensor  was  attached  to 
the  UAV  approximately  8  metres  below  the  actual  sensor  location  to  avoid  the  sensor 
responding  to  apparent  engine  heat  haze  from  the  VBS2  Scan  Eagle  model  and  obstructing 
viewing.  The  sensor  azimuth,  pitch  and  zoom  were  set  programmatically,  however,  the 
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sensor  roll  was  set  by  VBS2  automatically  to  keep  the  sensor  view  level  regardless  of  the 
UAV's  bank. 

Implementing  a  custom  sensor  model  required  very  frequent  updates  of  the  sensor  view 
due  to  VBS2  resetting  it  between  each  display  frame  update.  This  imposed  performance 
requirements  on  the  VBS  script  execution,  as  missing  a  sensor  view  update  between 
display  frames  would  result  in  the  sensor  view  flickering  between  two  views.  In  support 
of  this,  VBS2  initialisation  parameters  were  adjusted  to  prevent  script  interruption  and 
increase  the  available  time  for  script  execution. 

Further  to  the  sensor  model,  a  custom  heads  up  display  (HUD)  was  developed  utilising 
VBS2  resource  titles,  as  depicted  in  Figure  2.  The  HUD  extracted  data  from  variables 
stored  by  the  relevant  scripts,  including  scenario  control,  the  flight  model,  the  sensor 
model,  and  the  system  status  model. 

2.2.4. 1  Areas  of  investigation 

An  area  of  investigation  was  defined  as  the  surface  area  below  the  UAV  within  the 
perimeter  of  its  flight  path  as  indicated  on  the  mini  map  (see  Figure  2).  Areas  of 
investigation  were  designed  around  a  subset  of  representative  maritime  environments 
through  consultation  with  two  subject  matter  experts  (SMEs),  and  were  all  located  within 
a  single  VBS2  scenario.  When  a  change  of  area  was  required,  scripting  relocated  the  UAV 
to  the  new  area,  positioned  so  as  to  already  be  in  the  loiter  flight  pattern  with  configured 
bearings  and  directions  (clockwise  or  anti  clockwise). 

Within  an  area  of  investigation  was  a  vessel  of  interest  (VOI)  with  its  location  and 
coordinates  displayed  (see  Figure  2).  The  density  of  vessels  within  the  area  was  either  low 
(9-12  vessels  in  total)  or  high  (27-36  vessels  in  total).  The  vessels  for  each  area  were 
programmatically  created  and  removed  at  the  beginning  and  end  of  flight  in  the  area.  This 
was  to  ensure  the  participants  would  not  be  able  to  inadvertently  sight  vessels  from 
another  area. 

2. 2. 4.2  Reporting  task 

The  participants  were  required  to  provide  details  about  an  area  of  investigation  via  an 
investigation  report  provided  on  the  secondary  screen  (HP  20  inch  LP2065).  The  report 
included  details  about  the  VOI  as  well  as  the  total  number  of  vessels  in  the  area,  and  was 
designed  in  consultation  with  SMEs.  The  report  remained  constant  for  every  area  and 
details  were  entered  via  drop  down  menus  under  each  section  with  selections  made  via 
the  mouse,  or  via  direct  input  through  the  keyboard. 

Reports  were  implemented  externally  to  VBS2  utilising  a  specialised  programming 
language  known  as  Groovy.  To  support  coordination  with  VBS2,  a  file-based  event 
coordination  system  was  employed,  and  report  data  was  logged  to  csv  files.  Figure  3 
details  the  structure  of  the  report. 
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Report  layout _  Drop  down  menu 
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Figure  3:  Investigation  report  structure  and  operator  action  details. 


The  participants  had  an  additional  picture  matrix  which  was  attached  below  the 
secondary  screen  and  contained  pictures  of  all  the  vessels  used  in  the  simulation.  Each 
picture  was  columned  under  one  of  the  four  categories  available  to  choose  from  in  the 
drop  down  menu  under  ''Class/ Type". 

2.2.4. 3  System  status  task 

The  participants  were  also  required  to  respond  to  changes  in  the  system  status  indicators 
which  occurred  at  random  intervals  (see  Figure  2).  When  a  system  status  changed  from 
green  to  red,  a  corresponding  button  on  the  joystick  base  needed  to  be  pressed  in  order  to 
change  the  system  status  back  to  green.  If  the  corresponding  button  was  not  pressed,  the 
system  status  indicator  would  change  back  to  green  after  15  seconds. 

A  scripted  system  status  model  was  developed  that  pre-calculated  scheduled  status  failure 
times  from  the  beginning  of  each  task.  The  schedule  ensured  that  multiple  status  failures 
could  not  occur  concurrently,  and  failures  occurred  randomly  across  the  allowed  task  time 
with  a  uniform  distribution.  The  resource  titles  provided  by  the  HUD  were  utilised  to 
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display  each  system  status.  Custom  input  handling  with  the  script  allowed  for  status 
acknowledgements  from  participants. 

2.2 AA  Sensor  control 

Three  sensor  control  conditions  were  implemented: 

1.  Manual:  The  sensor  required  complete  manipulation  with  the  joystick  from 
participants  for  all  mission  tasks.  Participants  were  required  to  use  the  sensor 
direction  with  its  crosshair  coordinates  and  the  VOI  location  with  its  surface 
coordinates  to  search  for  the  VOI.  Participants  were  also  required  to  manipulate  the 
sensor  in  order  to  locate  and  count  the  total  number  of  vessels  in  the  area. 

2.  Fixed:  Sensor  processes  were  completely  automated.  In  this  mode  the  sensor 
initially  located  the  VOI  and  remained  zoomed  (xl5)  and  locked  on  it  for 
35  seconds.  The  sensor  then  panned  back  to  normal  zoom  setting  (xl)  and 
remained  at  a  30°  pitch  with  a  70°  azimuth.  All  remaining  vessels  in  the  area  were 
captured  in  the  sensor's  field  of  view  (FOV)  as  the  UAV  flew  on  its  designated 
path.i  No  sensor  manipulation  with  the  joystick  was  permitted. 

3.  Adaptable:  The  sensor  started  operation  in  fixed  mode.  However,  if  participants 
chose  to  operate  in  manual  mode  at  any  time  they  could  press  a  "mode"  button  on 
the  joystick  base  that  would  allow  this.  If  participants  wished  to  change  back  to 
fixed  mode,  they  could  do  so  by  pressing  the  "mode"  button  again.  Whenever 
changing  back  to  fixed  mode,  the  sensor  would  always  set  itself  at  the  xl  zoom 
level  with  30°  pitch  and  70°  azimuth.  Participants  could  switch  modes  as  often  as 
they  deemed  necessary. 

Table  1  outlines  sensor  control  condition  by  area  used  in  the  study.  Each  participant  took 
part  in  all  conditions,  and  the  sensor  control  condition  order  was  counterbalanced  across 
the  participant  pool  to  control  for  practice  effects. 


Table  1.  Sensor  control  condition  by  area 


Sensor  control 
condition 

Area  density 

Number  of  areas 

System  status 
changes  per  area 

Manual 

Low  (9-12  vessels)  and 

6  (3  low  followed  by 

Up  to  6 

high  (27-36  vessels) 

3  high  density) 

Fixed 

Low  (9-12  vessels)  and 

6  (3  low  followed  by 

Up  to  6 

high  (27-36  vessels 

3  high  density) 

Adaptable 

Low  (9-12  vessels)  and 

6  (3  low  followed  by 

Up  to  6 

high  (27-36  vessels 

3  high  density) 

^  The  flight  time  after  VOI  investigation  is  145  seconds,  which  was  based  on  observing  the  UAV 
complete  a  flight  lap  (see  2.3.2).  It  should  take  a  UAV  158  seconds  to  cover  this  distance  travelling  at 
a  speed  of  100  km/h  However,  the  UAV  was  observed  to  vary  its  speed  between  60  km/h  to 
120  km/h.  Researchers  should  be  aware  of  this  variability,  but  we  do  not  believe  this  affected  our 
study  outcomes. 
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For  the  reporting  task,  the  performance  measures  were  the  total  reporting  time,  which 
included  the  VOI  reporting  time  and  remaining  total  vessels  count  time,  as  well  as  the 
errors  in  the  VOI  detail  and  the  total  number  of  vessels  counted.  For  the  system  status 
task,  the  performance  measures  included  the  system  status  response  time  and  miss  rate. 
Additionally,  in  the  adaptable  sensor  control  condition,  the  number  of  changes  and  time 
between  manual  and  fixed  modes  were  captured. 

To  support  the  derivation  of  performance  measures,  the  data  logging  script  periodically 
polled  data  variables  for  storage  in  csv  files  for  subsequent  analysis  in  Microsoft  Excel. 
Separate  data  files  were  created  for  the  UAV,  vessels  and  system  status  due  to  different 
data  fields  and  logging  frequency  requirements.  All  automatic  data  logging,  with  the 
exception  of  the  operator  reports,  was  coordinated  from  this  script. 

2.2.5  Subjective  Workload 

Self-reported  workload  was  captured  using  the  NASA  Task  Load  Index  (NASA-TLX) 
(Hart  &  Staveland,  1988).  The  participants  rated  their  workload  across  six  subscales  from 
0-100  of  Mental  Demand,  Physical  Demand,  Temporal  Demand,  Performance,  Effort,  and 
Frustration.  Higher  overall  scores  indicated  higher  perceived  workload. 

2.2.6  Preference  ranking 

A  sensor  control  condition  preference  ranking  was  collected  post  the  experimental  session 
from  "1  -  Most  preferred"  to  "3  -  Least  preferred".  This  was  created  specifically  for  the 
current  study  to  gather  proportional  data  on  which  sensor  control  condition  was  most  and 
least  favoured  by  the  participants. 


2.3  Procedure 

After  obtaining  informed  consent,  the  participants  were  screened  for  any  visual  acuity  and 
colour  vision  deficiencies  and  demographic  data  were  obtained.  All  participants 
underwent  a  structured  training  session  followed  by  the  experimental  session. 

2.3.1  Training 

All  training  was  provided  in  a  systematic  fashion  with  all  elements  of  the  simulation  and 
their  functions  explained.  A  training  area  allowed  participants  to  practice  using  the 
elements  of  the  simulation  as  well  as  practice  their  mission  tasks  prior  to  the  experimental 
session.  The  training,  including  all  function  explanations  and  practice  time,  took 
approximately  20  minutes. 
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2.3.2  Experimental  session 

The  experimental  session  required  participants  to  find,  investigate,  and  report  details 
about  the  VOI  in  an  area  of  investigation.  Following  this,  the  participants  were  required  to 
find  and  count  the  total  number  of  vessels  in  the  area,  but  priority  was  always  given  to 
reporting  the  details  about  the  VOI.  While  on  any  mission  task,  the  participants  were  to 
also  acknowledge  any  system  status  changes  that  occurred  as  quickly  as  possible.  The 
participants  were  cued  via  a  three  second  count  down  before  both  screens  would  activate 
and  the  UAV  would  be  in  flight  in  the  area.  They  had  a  maximum  of  180  seconds  to 
complete  all  the  tasks  before  the  screens  would  go  blank  again. 

Those  taking  part  were  instructed  to  complete  all  tasks  as  fast,  but  as  accurately  as 
possible.  If  they  completed  the  tasks  prior  to  the  maximum  time  given,  they  could  click 
''Submit"  on  the  investigation  report,  which  would  end  the  time  in  the  area  and  all  screens 
would  go  blank.  If  they  did  not  click  "Submit"  the  time  remaining  in  the  area  would  count 
down  to  zero  and  all  screens  would  subsequently  go  blank,  but  any  information  they  had 
inputted  on  the  investigation  report  would  be  saved  by  the  system. 

After  each  block  of  three  areas  of  investigation,  the  simulation  was  paused  and  the 
participants  were  asked  to  fill  out  their  ratings  of  current  workload  as  indicated  by  the 
dimensions  on  the  NASA-TLX  (Hart  &  Staveland,  1988).  This  process  was  repeated  until 
completion  of  the  simulation.  Participants  were  always  told  which  sensor  condition  they 
would  operate  under,  however  no  indication  was  made  as  to  the  differing  area  density. 

After  the  completion  of  the  experimental  session,  the  participants  were  asked  to  rank  their 
preferences  of  each  of  the  sensor  control  conditions.  The  experimental  session  took 
approximately  60  minutes,  thus  the  total  participation  time  with  training  was 
approximately  80  minutes. 
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3.  Results 


3.1  Descriptive  statistics 

Participants  gaming  experience  over  the  past  4  weeks  ranged  from  1  (Not  at  all)  to  5  (  7  or 
more  hours  per  week),  but  were  generally  non-gamers  (M  =  2.21,  SD  =  1.38). 


3.2  Reporting  task 

3.2.1  Total  reporting  time 

A  3  (manual,  fixed,  and  adaptable  sensor  control)  x  2  (low  and  high  area  density)  repeated 
measures  ANOVA  was  used  to  compare  total  reporting  times  across  conditions. 
Mauchly's  test  indicated  the  assumption  of  sphericity  had  been  violated  for  the  main  effect 
of  sensor  condition  (7^(2)  =  9.12,  p  =  .010)  and  interaction  effect  between  sensor  condition 
and  density  {y}{2)  =  10.15,  p  =  .006)  therefore  degrees  of  freedom  were  corrected  using 
Greenhouse-Geisser  estimates  of  sphericity  (e=  .82  and  .80  respectively).  Table  2  shows 
the  descriptive  statistics  of  total  reporting  times  for  sensor  conditions  and  area  densities 
overall. 

Table  2.  Overall  total  reporting  times  across  sensor  condition  and  area  density 


Sensor  condition 

Mean  (seconds) 

SE 

Manual 

168.43 

2.05 

Fixed 

170.53 

2.34 

Adaptable 

Density 

152.09 

4.27 

Low 

160.82 

2.69 

High 

166.55 

1.12 

There  was  a  significant  main  effect  of  sensor  control  condition  on  total  reporting  times 
(f(1.63,  60.47)  =  17.01,  p  <  .001,  partial  r}'^  =  .32).  Total  reporting  times  were  faster  in 
adaptable  compared  to  both  fixed  and  manual  sensor  control  (f(37)  =  4.56,  p  <  .001,  r  =  .60 
and  f(37)  =  4.56,  p  <  .001,  r  =  .60  respectively),  and  both  were  large  effects.  This  suggested 
adaptable  automation  improved  the  performance  of  this  task  compared  to  no  automation. 
Total  reporting  times  did  not  significantly  differ  between  manual  and  fixed  sensor  control 
(f(37)  =  -.81,  p  =  .421,  r  =  .13),  which  indicated  fixed  automation  did  not  improve  the 
performance  of  this  task  compared  to  no  automation. 

There  was  also  a  significant  main  effect  of  area  density  on  total  reporting  times  (F(l,  37)  = 
15.65,  p  <  .001,  partial  rj^  =  .30)  with  total  reporting  times  faster  in  low  compared  to  high 
density,  and  this  was  a  large  effect. 

There  was  no  significant  interaction  effect  between  sensor  control  condition  and  area 
density  with  regard  to  total  reporting  times  (f(1.61,  59.40)  =  1.81,  p  =  .306,  partial  =  .03). 
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Figure  4  shows  the  total  reporting  time  against  sensor  condition  for  low  and  high  area 
density. 


Manual  Low  Manual  High  Fixed  Low^  Fixed  High  Adaptable  Low  Adaptable  High 

Condition 


Figure  4:  Total  reporting  time  against  sensor  condition  for  low  and  high  area  density.  Error  bars 
indicate  95%  confidence  intervals. 


3.2.2  Vessel  of  Interest  reporting  time 

A  3  (manual,  fixed,  and  adaptable  sensor  control)  x  2  (low  and  high  area  density)  repeated 
measures  Analysis  of  Variance  (ANOVA)  was  used  to  compare  VOI  reporting  times  across 
conditions.  Mauchly's  test  indicated  the  assumption  of  sphericity  had  been  violated  for  the 
main  effect  of  sensor  condition  =  9.20,  p  =  .010),  therefore  degrees  of  freedom  were 
corrected  using  Greenhouse-Geisser  estimates  of  sphericity  (e=  .82).  Table  3  shows  the 
descriptive  statistics  of  VOI  reporting  times  for  sensor  conditions  and  area  densities 
overall. 

Table  3.  Overall  VOI  reporting  times  across  sensor  condition  and  area  density 


Sensor  condition 

Mean  (seconds) 

SE 

Manual 

98.80 

3.38 

Fixed 

39.58 

1.58 

Adaptable 

Density 

51.99 

2.84 

Low 

59.85 

2.11 

High 

67.07 

1.98 

There  was  a  significant  main  effect  of  sensor  control  condition  on  VOI  reporting  times 
(f  (1.63,  60.38)  =  144.44,  p  <  .001,  partial  ?j2  =  .80).  VOI  reporting  times  were  faster  in  both 
fixed  and  adaptable  compared  to  manual  sensor  control  (f(37)  =  17.21,  p  <  .001,  r  =  .94  and 


UNCLASSIFIED 


19 


UNCLASSIFIED 

DST-Group-TR-2953 

t{37)  =  10.55,  p  <  .001,  r  =  .87  respectively),  and  both  were  very  large  effects.  This  indicated 
both  fixed  and  adaptable  automation  improved  the  performance  of  this  task  over  no 
automation.  VOI  reporting  times  were  also  faster  in  fixed  compared  to  adaptable  sensor 
control  (f(37)  =  -4.14,  p  <  .001,  r  =  .56)  and  this  was  a  large  effect. 

There  was  also  a  significant  main  effect  of  area  density  on  VOI  reporting  times  (F(l,  37)  = 
9.74,  p  =  .003,  partial  =  .21)  with  VOI  reporting  times  faster  in  low  compared  to  high 
density,  and  this  was  a  large  effect. 

There  was  no  significant  interaction  effect  between  sensor  control  condition  and  area 
density  with  regard  to  VOI  reporting  times  (F(2,  74)  =  .73,  p  =  .486,  partial  =  .02).  Figure 
5  shows  the  VOI  reporting  time  against  sensor  condition  for  low  and  high  area  density. 


VOI  reporting  times 

180  n - 

160 - 

140 - 


Manual  Low  Manual  High  Fixed  Low^  Fixed  High  Adaptable  Low  Adaptable  High 

Condition 


Figure  5:  VOI  reporting  time  against  sensor  condition  for  low  and  high  area  density.  Error  bars 
indicate  95%  confidence  intervals. 


3.2.3  Total  vessels  count  time 

A  3  (manual,  fixed,  and  adaptable  sensor  control)  x  2  (low  and  high  area  density)  repeated 
measures  ANOVA  was  used  to  compare  the  total  time  searching  and  counting  the  total 
number  of  vessels  in  areas  across  conditions.  This  search  time  was  calculated  by  removing 
the  VOI  reporting  time  from  the  total  reporting  time.  Mauchly's  test  indicated  the 
assumption  of  sphericity  had  been  violated  for  the  interaction  between  sensor  condition 
and  density  (7^(2)  =  12.43,  p  =  .002),  therefore  degrees  of  freedom  were  corrected  using 
Greenhouse-Geisser  estimates  of  sphericity  (e=  .77).  Table  4  shows  the  descriptive 
statistics  of  total  vessels  count  time  for  sensor  conditions  and  area  densities  overall. 
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Table  4.  Overall  total  vessels  count  times  across  sensor  condition  and  area  density 


Sensor  condition 

Mean  (seconds) 

SE 

Manual 

70.08 

3.05 

Fixed 

130.97 

2.74 

Adaptable 

Density 

100.09 

3.79 

Low 

101.26 

2.61 

High 

99.49 

2.52 

There  was  a  significant  main  effect  of  sensor  condition  on  total  vessel  count  time  (F(2,  74)  = 
110.78,  p  <  .001,  partial  rf-  =  .75).  Total  vessel  count  times  were  faster  in  manual  compared 
to  both  fixed  and  adaptable  sensor  control  (f(37)  =  -15.26,  p  <  .001,  r  =  .93  and  t{37)  =  -7.03, 
p  <  .001,  r  =  76  respectively),  and  both  were  very  large  effects.  Total  vessel  count  times 
were  also  faster  in  adaptable  compared  to  fixed  sensor  control  (f(37)  =  7.71,  p  <  .001,  r  = 
.79),  and  this  was  also  a  very  large  effect.  It  should  be  noted  that  these  times  are  influenced 
by  the  time  spent  on  the  VOI  task,  which  was  directed  to  be  the  highest  priority  for 
participants,  and  the  overall  constraint  of  both  tasks  being  completed  within  180  seconds. 

There  was  no  significant  main  effect  of  density  on  total  vessel  time  (F(l,  37)  =  .45,  p  =  .507, 
partial  r}^  =  .01),  nor  any  significant  interaction  effect  between  sensor  condition  and  area 
density  (F(1.55,  57.28)  =  1.11,  p  =  .325,  partial  r}^  =  .03).  Figure  6  shows  the  total  vessel 
count  times  against  sensor  condition  for  low  and  high  density. 
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Figure  6:  Total  vessels  count  times  against  sensor  conditions  for  low  and  high  area  density.  Error 
bars  indicate  95%  confidence  intervals. 

3.2.4  VOI  reporting  error 

A  3  (manual,  fixed,  and  adaptable  sensor  control)  x  2  (low  and  high  area  density)  repeated 
measures  ANOVA  was  used  to  compare  VOI  reporting  errors  across  conditions. 
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Mauchly's  test  indicated  the  assumption  of  sphericity  had  been  violated  for  the  main  effect 
of  sensor  condition  {y^{2)  =  22.99,  p  <  .001)  and  main  effect  of  density  (7^(2)  =  6.82,  p  =  .033) 
therefore  degrees  of  freedom  were  corrected  using  Greenhouse-Geisser  estimates  of 
sphericity  (e=  .68  and  .85  respectively).  Table  5  shows  descriptive  statistics  of  VOI 
reporting  errors  for  sensor  conditions  and  area  densities  overall.  Error  was  calculated  and 
termed  in  percentages  to  provide  meaningful  comparison  between  conditions. 

Table  5.  Overall  VOI  reporting  error  across  sensor  condition  and  area  density 


Sensor  condition 

Mean  error  (%) 

SE 

Manual 

20.4 

2.2 

Fixed 

8.6 

.8 

Adaptable 

Density 

11.1 

1.1 

Low 

12.6 

1.1 

High 

14.1 

1.2 

There  was  a  significant  main  effect  of  sensor  control  condition  on  VOI  reporting  error 
(F(1.36,  50.28)  =  21.45,  p  <  .001,  partial  rf-  =  .37).  VOI  reporting  error  was  greatest  in  manual 
compared  to  both  fixed  and  adaptable  sensor  control  and  (f(37)  =  5.47,  p  <  .001,  r  =  .67  and 
f(37)  =  4.14,  p  <  .001,  r  =  .56  respectively),  and  both  were  large  effects.  Similar  to  VOI 
reporting  time,  this  suggested  fixed  and  adaptable  automation  improved  the  performance 
on  this  task  compared  to  no  automation.  VOI  reporting  error  did  not  significantly  differ 
between  fixed  and  adaptable  sensor  control  (f(37)  =  -2.37,  p  =  .023,  r  =  .36;  Bonferroni 
correction  required  p  <  .017). 

There  was  no  significant  main  effect  of  area  density  on  VOI  reporting  error  (F(l,  37)  =  3.11, 
p  =  .086,  partial  rf-  =  .08),  nor  any  significant  interaction  effect  between  sensor  control 
condition  and  area  density  (F(2,  74)  =  1.01,  p  <  .368,  partial  rf-  =  .03).  Figure  7  shows  the 
VOI  reporting  error  against  sensor  condition  for  low  and  high  area  density. 
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Figure  7:  VOI  reporting  error  against  sensor  condition  for  low  and  high  area  density.  Error  bars 
indicate  95%  confidence  intervals. 


3.2.5  Total  vessels  count  error 

A  3  (manual,  fixed,  and  adaptable  sensor  control)  x  2  (low  and  high  area  density)  repeated 
measures  ANOVA  was  used  to  compare  total  vessels  count  errors  across  conditions.  Table 
6  shows  the  descriptive  statistics  of  errors  in  the  total  number  of  vessels  counted  for  sensor 
conditions  and  area  densities  overall.  Total  vessels  count  error  was  calculated  and  termed 
in  percentages  to  provide  meaningful  comparison  between  conditions. 

Table  6.  Overall  error  in  the  total  number  of  vessels  in  area  counted  across  sensor  condition  and 
area  density 


Sensor  condition 

Mean  error  (%) 

SE 

Manual 

34.8 

3.8 

Fixed 

20.8 

1.8 

Adaptable 

Density 

23.1 

2.2 

Low 

26.9 

2.1 

High 

25.5 

2.1 

There  was  a  significant  main  effect  of  sensor  control  condition  on  error  in  the  total  number 
of  vessels  counted  (F(2,  74)  =  10.32,  p  <  .001,  partial  rf  =  .22).  The  error  in  the  total  number 
of  vessels  counted  was  greater  in  manual  compared  to  both  fixed  and  adaptable  sensor 
control  (f(37)  =  3.77,  p  =  .001,  r  =  .53  and  t{37)  =  3.36,  p  =  .002,  r  =  .48  respectively),  and  both 
were  large  effects.  Similar  to  total  reporting  time,  this  indicated  adaptable  automation 
improved  performance  on  this  task  compared  to  no  automation.  However,  mixed  results 
were  apparent  comparing  fixed  automation  to  no  automation  on  this  task.  While  fixed 
automation  improved  performance  with  regard  to  the  error  in  the  total  number  of  vessels 
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counted,  the  total  time  to  complete  the  investigation  report  did  not  differ  between  the 
fixed  and  manual  sensor  control  conditions  (see  3.2.2).  The  total  vessels  count  error  did  not 
significantly  differ  between  fixed  and  adaptable  sensor  control  (f(37)  =  -.88,  p  =  .384,  r  = 
.14).  Improved  total  reporting  time  with  no  increase  in  error  on  either  VOI  reporting  or  the 
total  number  of  vessels  counted  suggested  adaptable  automation  reduced  response  time 
without  degrading  accuracy  compared  to  fixed  automation. 

There  was  no  significant  main  effect  of  area  density  on  error  in  the  total  number  of  vessels 
counted  (F(l,  37)  =  .69,  p  =  .411,  partial  rj^  =  .02),  however  there  was  a  significant 
interaction  effect  between  sensor  control  condition  and  area  density  (F(2,  74)  =  7.00,  p  = 
.002,  partial  r}^  =  .16).  This  indicated  that  the  sensor  condition  had  different  effects  on  total 
vessels  count  error  depending  on  density.  Error  in  the  total  number  of  vessels  counted  was 
significantly  greater  in  manual  control  in  high  compared  to  low  density  (f(37)  =  -2.29,  p  = 
.006,  r  =  .35),  but  greater  in  fixed  control  in  low  compared  to  high  density  (f(37)  =  2.86,  p  = 
.007,  r  =  .43),  and  both  were  medium  to  large  effects.  This  trend  was  the  same  as  the  latter 
in  adaptable  control,  although  the  difference  was  not  significant  (f(37)  =  1.11,  p  =  .273,  r  = 
.18).  Figure  8  shows  the  error  in  the  total  number  of  vessels  counted  against  sensor 
condition  for  low  and  high  area  density. 
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Figure  8:  Error  in  the  total  number  of  vessels  counted  against  sensor  condition  for  low  and  high 
area  density.  Error  bars  indicate  95%  confidence  intervals. 

3.2.6  System  status  response  time 

A  3  (manual,  fixed,  and  adaptable  sensor  control)  x  2  (low  and  high  area  density)  repeated 
measures  ANOVA  was  used  to  compare  system  status  response  times  across  conditions. 
Table  7  shows  descriptive  statistics  of  system  status  response  times  for  sensor  conditions 
and  area  densities  overall. 
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Table  7.  Overall  system  status  response  times  across  sensor  condition  and  area  density 


Sensor  condition 

Mean  (seconds) 

SE 

Manual 

4.33 

.49 

Fixed 

3.48 

.44 

Adaptable 

Density 

4.37 

.44 

Low 

4.31 

.40 

High 

3.81 

.37 

There  was  a  significant  main  effect  of  sensor  control  condition  on  system  status  response 
times  (F(2,  74)  =  4.478,  p  =  .015,  partial  rf-  =  .11).  System  status  response  times  were  fastest 
in  fixed  compared  to  both  manual  and  adaptable  sensor  control  (f(37)  =  2.79,  p  =  .008,  r  = 
.42  and  t{37)  =  -2.60,  p  =.013,  r  =  .39  respectively),  and  these  were  both  medium  to  large 
effects,  which  suggested  fixed  automation  improved  performance  of  this  task  compared  to 
no  automation.  Response  times  did  not  significantly  differ  between  manual  and  adaptable 
sensor  control  (f(37)  =  -.12,  p  =  .906,  r  =  .03),  which  indicated  adaptable  automation  did  not 
improve  performance  on  this  task  compared  to  no  automation. 

There  was  a  significant  main  effect  of  area  density  on  system  status  response  times  (F(l, 
37)  =  19.445,  p  <  .001,  partial  =  .34)  with  system  status  response  times  faster  in  high 
compared  to  low  density,  and  this  was  a  large  effect. 

There  was  no  significant  interaction  effect  between  sensor  control  condition  and  area 
density  with  regard  to  system  status  response  times  (F(2,  74)  =  1.35,  p  =  .266,  partial  = 
.04).  Figure  9  shows  system  status  response  time  against  sensor  condition  for  low  and  high 
area  density. 
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Figure  9:  System  status  response  time  against  sensor  condition  for  low  and  high  area  density.  Error 
bars  indicate  95%  confidence  intervals. 
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3.2.7  System  status  miss  rates 

A  3  (manual,  fixed,  and  adaptable  sensor  control)  x  2  (low  and  high  area  density)  repeated 
measures  ANOVA  was  used  to  compare  system  status  miss  rates  across  conditions. 
Table  8  shows  descriptive  statistics  of  system  status  miss  rates  for  sensor  conditions  and 
area  densities  overall.  Miss  rates  were  calculated  and  termed  in  percentages  to  provide 
meaningful  comparison  between  conditions 

Table  8.  Overall  system  status  miss  rates  across  sensor  condition  and  area  density 


Sensor  condition 

Mean  error  (%) 

SE 

Manual 

12.9 

3.2 

Fixed 

8.5 

1.9 

Adaptable 

Density 

11.4 

2.7 

Low 

12.7 

2.4 

High 

9.2 

2.2 

There  was  no  significant  main  effect  of  sensor  control  condition  on  system  status  miss 
rates  (F(2,  74)  =  1.74,  p  =  .183,  partial  rf-  =  .05),  nor  any  significant  interaction  effect 
between  sensor  control  condition  and  area  density  with  regard  to  system  status  miss  rates 
(F(2,  74)  =  2.23  p  =  .115,  partial  =  .06). 

However,  there  was  a  significant  main  effect  of  area  density  on  system  status  miss  rate 
(F(l,  37)  =  14.43,  p  =  .001,  partial  rf-  =  .28).  System  status  miss  rates  were  lower  in  high 
compared  to  low  density,  and  this  was  a  large  effect.  Figure  10  shows  the  system  status 
miss  rate  against  sensor  condition  for  low  and  high  area  density. 
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Figure  10:  System  status  miss  rate  against  sensor  condition  for  low  and  high  area  density.  Error 
bars  indicate  95%  confidence  intervals. 
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3.3  Adaptable  sensor  control 

For  the  adaptable  sensor  condition  only,  the  number  of  mode  changes  as  well  as 
differences  in  time  spent  in  the  different  modes  was  examined  for  low  and  high  area 
density.  The  time  spent  in  the  different  modes  was  then  analysed  separately  for  VOI 
reporting  and  total  number  of  vessels  reporting. 

3.3.1  Mode  changes 

A  paired  t-test  was  used  to  compare  the  number  of  mode  changes  under  adaptable  sensor 
control  between  low  and  high  area  density.  The  number  of  mode  changes  did  not 
significantly  differ  between  low  (M  =  2.32,  95%,  SE  =  .29)  and  high  (M  =  2.00,  95%,  SE  = 
.19)  density  {t  (37)  =  1.20,  p  =  .237,  r  =  .20),  but  this  indicated  the  participants  made  use  of 
mode  changes  in  both  low  and  high  density. 

3.3.2  Mode  time 

A  2  X  2  ANOVA  was  used  to  compare  the  amounts  of  time  spent  in  manual  and  fixed 
modes  under  adaptable  sensor  control  in  low  and  high  area  density.  Table  9  shows  the 
descriptive  statistics  of  amount  of  time  spent  in  each  mode  under  adaptable  control  and 
area  densities  overall. 

Table  9.  Overall  time  spent  in  each  mode  in  the  adaptable  sensor  control  condition 


Adaptable  mode 

Mean  (seconds) 

SE 

Manual 

89.21 

4.38 

Fixed 

60.18 

5.05 

Density 

Low 

73.18 

2.37 

High 

76.21 

2.08 

There  was  a  significant  main  effect  of  adaptable  sensor  control  mode  on  time  spent  (F(l, 
37)  =  11.81,  p  =  .001,  partial  =  .24),  where  more  time  was  spent  in  manual  compared  to 
fixed  mode,  and  this  was  a  large  effect. 

There  was  a  significant  main  effect  of  area  density  on  adaptable  sensor  control  mode  (F(l, 
37)  =  5.00,  p  =  .031,  partial  =  .12).  Nonetheless,  examining  time  spent  in  each  mode  was 
not  useful  when  these  times  were  combined  into  an  overall  time  each  for  low  and  high 
density.  Subsequently,  this  result  presented  no  further  interest  for  discussion. 

There  was  no  significant  interaction  effect  between  adaptable  sensor  control  mode  and 
area  density  (F(l,  37)  =  .68,  p  =  .414,  partial  r}^  =  .02).  This  indicated  that  control  mode  did 
not  have  different  effects  depending  on  density,  and  fixed  automation  was  not  relied  upon 
more  in  higher  workload.  Figure  11  shows  the  times  spent  in  the  two  modes  for  low  and 
high  area  density  under  adaptable  sensor  control. 
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Figure  11:  Time  spent  in  each  mode  for  low  and  high  area  density  under  adaptable  sensor  control 
Error  bars  indicate  95%  confidence  intervals. 


3. 3. 2.1  Vessel  of  Interest  mode  time 

A  2  X  2  ANOVA  was  used  to  compare  the  amount  of  time  spent  between  manual  and  fixed 
modes  under  adaptable  sensor  control  during  VOI  reporting  time.  Table  10  shows  the 
descriptive  statistics  of  the  time  spent  in  each  mode  under  adaptable  control  and  area 
densities  overall  during  VOI  reporting. 


Table  10.  Overall  time  spent  in  each  mode  in  the  adaptable  sensor  control  condition  during  VOI 
reporting 


Adaptable  mode 

Mean  (seconds) 

SE 

Manual 

17.18 

2.45 

Fixed 

38.88 

1.04 

Density 

Low 

24.88 

1.97 

High 

27.18 

1.95 

There  was  a  significant  main  effect  of  adaptable  sensor  control  on  the  amount  of  time 
spent  between  the  modes  during  VOI  reporting  (F(l,  37)  =  49.53,  p  <  .001,  partial  rj^  =  .57) 
where  more  time  was  spent  in  fixed  compared  to  manual  mode,  and  this  was  a  very  large 
effect. 

There  was  no  significant  main  effect  of  area  density  on  amount  of  time  spent  between 
modes  (F(l,  37)  =  .70,  p  =  .407,  partial  =  .02),  and  this  result  did  not  warrant  further 
discussion  (see  3.3.2).  There  was  also  no  significant  interaction  effect  between  amount  of 
time  spent  between  modes  and  density  F(l,  37)  =  1.38,  p  =  .248,  partial  r}^  =  .04),  which 
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suggested  that  fixed  automation  was  not  relied  upon  more  in  higher  workload  for  this 
task. 

33.2.2  Total  vessels  mode  time 

A  2  X  2  ANOVA  was  used  to  compare  the  amount  of  time  spent  between  manual  and  fixed 
modes  under  adaptable  sensor  control  during  searching  and  counting  the  total  number  of 
vessels  in  the  area.  Table  11  shows  the  descriptive  statistics  of  the  time  spent  in  each  mode 
in  adaptable  control  and  area  densities  overall  during  total  number  of  vessels  searching 
and  counting. 

Table  11.  Overall  time  spent  in  each  mode  under  adaptable  sensor  control  during  total  number  of 
vessels  searching  and  counting 


Adaptable  mode 

Mean  (seconds) 

SE 

Manual 

71.25 

3.71 

Fixed 

24.72 

4.61 

Density 

Low 

47.54 

2.32 

High 

48.43 

2.51 

There  was  a  significant  main  effect  of  adaptable  sensor  control  on  the  amount  of  time 
spent  between  modes  during  total  number  of  vessels  searching  and  counting  (F(l,  37)  = 
38.62,  p  <  .001,  partial  =  .51)  where  more  time  was  spent  in  manual  compared  to  fixed 
mode,  and  this  was  a  very  large  effect. 

There  was  no  significant  main  effect  of  area  density  on  amount  of  time  spent  between 
modes  (F(l,  37)  =  .09,  p  =  .773,  partial  rj^  <  .01),  and  this  result  did  not  warrant  further 
discussion  (see  3.3.2).  There  was  also  no  significant  interaction  effect  between  amount  of 
time  spent  between  modes  and  density  (F(l,  37)  =  .02,  p  =  .902,  partial  r}^  <  .01),  which 
suggested  that  fixed  automation  was  not  relied  upon  more  in  higher  workload  for  this 
task. 


3.4  Subjective  workload 

A  3  (manual,  fixed,  and  adaptable  sensor  control)  x  2  (low  and  high  area  density)  repeated 
measures  ANOVA  was  used  to  compare  subjective  workloads  (NASA-TLX)  across 
conditions.  Table  12  shows  the  descriptive  statistics  of  perceived  workloads  for  sensor 
conditions  and  low  and  high  area  density  overall. 
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Table  12.  Overall  perceived  workload  across  sensor  condition  and  area  density 


Adaptable  mode 

Workload  score 

SE 

Manual 

58.52 

2.31 

Fixed 

42.27 

2.58 

Adaptable 

Density 

52.51 

2.34 

Low 

48.59 

2.06 

High 

53.61 

2.18 

There  was  a  significant  main  effect  of  sensor  control  condition  on  subjective  workload  (F(2, 
74)  =  30.20,  p  <  .001,  partial  =  .45).  Subjective  workload  was  higher  under  manual 
compared  to  both  fixed  and  adaptable  sensor  control  (f(37)  =  7.17,  p  <  .001,  r  =  .76  and  f(37) 
=  3.12,  p  =  .003,  r  =  .46  respectively),  and  both  were  large  effects.  Subjective  workload  was 
also  higher  under  adaptable  compared  to  fixed  sensor  control  (f(37)  =  -4.79,  p  <  .001,  r  = 
.62),  and  this  was  also  a  large  effect. 

There  was  a  significant  main  effect  of  area  density  on  subjective  workload  (F(l,  37)  =  34.08, 
p  <  .001,  partial  =  .48),  where  subjective  workload  was  greater  under  high  compared  to 
low  density,  and  this  was  a  very  large  effect. 

There  was  no  significant  interaction  effect  between  sensor  control  condition  and  area 
density  with  regard  to  subjective  workload  (F(2,  74)  =  .02,  p  =  .984,  partial  r}^  <  .01).  Figure 
12  shows  the  overall  workload  rating  against  sensor  condition  for  low  and  high  area 
density. 


Overall  subjective  workload 


Condition 


Figure  12:  Overall  workload  rating  against  sensor  condition  for  low  and  high  area  density.  Error 
bars  indicate  95%  confidence  intervals. 
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3.5  Sensor  control  condition  preferences 

Table  13  shows  the  distribution  of  preference  rankings  for  manual,  fixed,  and  adaptable 
sensor  control. 


Table  13.  Distribution  of  preference  rankings  for  manual,  fixed,  and  adaptable  sensor  control 


Sensor  condition 

1=‘  rank  (%) 

2nd  rank  (%) 

3rd  rank  (%) 

Manual 

7.9 

55.3 

36.8 

Fixed 

2.6 

39.5 

57.9 

Adaptable 

89.5 

5.3 

5.3 

Rankings  suggest  that  the  adaptable  sensor  control  was  more  preferred  compared  to 
manual  or  fixed  sensor  control.  A  greater  proportion  of  participants  ranked  manual  rather 
than  fixed  sensor  control  as  their  second  preference,  and  fixed  sensor  control  was  the  least 
preferred  for  the  majority  of  participants. 
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4.  Discussion 

The  aim  of  this  study  was  to  compare  the  performances  of  operators  on  a  selection  of  UAV 
tasks  under  adaptable,  fixed,  and  manual  sensor  control.  The  effect  of  automation  on 
performance  in  different  area  densities  was  also  investigated.  Four  hypotheses  were 
developed  to  address  this  aim.  The  first  hypothesis  was  that  fixed  automation  would 
improve  performance  on  tasks  compared  to  no  automation.  This  was  partially  supported, 
as  it  was  shown  that  fixed  sensor  control  improved  response  times  and  reduced  error  in 
VOI  reporting  compared  to  manual  sensor  control.  With  regard  to  total  reporting  response 
times,  however,  no  difference  was  found/ observed  between  fixed  and  manual  sensor 
control,  although  there  were  fewer  errors  in  reporting  the  total  number  of  vessels  under 
fixed  compared  to  manual  sensor  control.  Regarding  the  system  status  task,  it  was  shown 
that  fixed  sensor  control  improved  response  times  compared  to  manual  sensor  control, 
although  no  difference  existed  between  these  conditions  for  system  status  miss  rates. 

The  second  hypothesis  was  that  adaptable  automation  would  improve  performance  on 
tasks  compared  to  no  automation.  This  was  also  partially  supported,  as  it  was  shown  that 
adaptable  control  improved  response  times  and  reduced  the  error  rates  in  identifying  VOI 
and  total  reporting  compared  to  manual  control.  However,  regarding  the  system  status 
task,  response  times  and  miss  rates  did  not  differ  between  adaptable  and  manual  sensor 
control. 

The  third  hypothesis  was  that  an  accuracy  over  speed  trade  off  would  exist  under 
adaptable  compared  to  fixed  automation,  where  accuracy  would  improve  at  the  expense 
of  longer  response  times  in  adaptable  compared  to  fixed  automation.  However,  this  was 
not  supported.  Although  adaptable  sensor  control  was  shown  to  increase  VOI  reporting 
times  compared  to  fixed  sensor  control,  no  difference  in  VOI  error  existed  between 
conditions.  Adaptable  sensor  control  actually  reduced  total  reporting  time  compared  to 
fixed  sensor  control,  yet  no  difference  in  error  rates  in  reporting  the  total  number  of 
vessels  existed  between  these  conditions  either.  Regarding  the  system  status  task,  response 
times  were  longer  in  adaptable  compared  to  fixed  sensor  control,  but  no  difference  existed 
between  these  conditions  for  system  status  miss  rates. 

Finally,  it  was  hypothesised  that  participants  would  rely  on  fixed  automation  more  when 
under  higher  workload,  but  this  also  was  not  supported.  Under  adaptable  sensor  control, 
the  time  spent  in  fixed  mode  was  not  found  to  differ  between  low  and  high  area  density, 
nor  was  it  found  to  differ  for  manual  mode.  Overall,  participants  actually  spent  more  time 
in  manual  compared  to  fixed  mode,  but  when  this  was  broken  down  further  it  was 
revealed  that  whether  more  time  was  spent  in  manual  or  fixed  mode  depended  on  the 
type  of  reporting  task  being  performed  (see  4.4). 


4.1  Fixed  automation 

While  the  tasking  was  different  in  the  current  study,  the  finding  that  a  fixed  LOA 
improved  performance  across  reporting  and  system  status  tasks  is  in  line  with  current 
research  (Dixon  et  al.,  2005;  Prinet  et  al.,  2012;  Ruff  et  al.,  2002;  Wickens  &  Dixon,  2002). 
Additionally,  implementing  a  fixed  LOA  was  perceived  as  having  less  workload  overall 
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compared  to  when  no  automation  was  used.  In  this  context,  it  was  probably  unlikely  that 
VOI  reporting  would  take  much  longer  than  the  allotted  time  of  35  seconds  that  the  sensor 
was  locked  and  zoomed  in  on  the  VOI  in  the  fixed  sensor  control  condition.  There  was  no 
option  to  lengthen  the  time  the  sensor  was  focused  on  the  VOI,  and  the  findings  suggested 
that  participants  did  complete  this  task  accordingly,  using  an  extra  four  to  five  seconds 
after  the  sensor  had  moved  off  the  VOI  to  finish  filling  out  the  VOI  report  section,  with  less 
than  10  percent  error  on  average.  Without  automated  assistance,  VOI  reporting  time 
increased  under  manual  control.  This  result  was  consistent  with  trends  found  by  Prinet  et 
al.  (2012),  where  participants  also  had  the  task  demand  of  having  to  locate  the  VOI  before 
reporting  could  commence  and  so  the  response  time  on  that  task  was  poor.  The  error  in 
VOI  reporting  was  also  greater  in  manual  compared  to  fixed  sensor  control,  which  may 
have  been  due  to  the  lack  of  automated  assistance  adding  time  pressure  affecting 
performance  on  this  primary  task  (Miller,  Pelican,  &  Goldman,  2000),  or  potential 
difficulties  in  manually  adjusting  the  sensor  to  get  a  reasonable  view  of  the  VOI  to  obtain 
the  required  details  about  it.  Additionally,  the  participants  may  have  at  times  located  a 
different  vessel  and  reported  it  as  the  VOI,  which  would  have  increased  the  error  in  the 
vessel  of  interest  reporting  for  this  task,  as  an  artefact  of  the  simulation  ensured  details 
about  the  VOI  were  never  completely  duplicated  on  another  vessel.  These  findings  suggest 
that  for  the  purpose  of  VOI  reporting,  a  fixed  LOA  on  sensor  control  to  assist  operators 
reduces  reporting  times  and  errors. 

The  highest  priority  task  for  participants  was  reporting  on  the  VOI.  Therefore,  under 
manual  sensor  control  the  time  available  to  count  the  total  number  of  vessels  was  the  time 
remaining  after  VOI  reporting  was  complete,  but  for  fixed  automation  this  was  set  at 
145  seconds.  Most  participants  took  close  to  the  full  time  to  complete  both  VOI  reporting 
and  vessel  counting.  Therefore,  for  manual  and  adaptable  sensor  control  the  vessel  count 
time  may  reflect  the  remaining  time  available  for  this  task,  as  opposed  to  the  time  that 
would  have  been  taken  had  an  overall  time  constraint  not  been  imposed.  Under  manual 
sensor  control,  over  half  of  the  available  time  in  the  area  was  used  to  search  and  report  on 
the  VOI.  Therefore,  nearly  all  of  the  remaining  time  (70  seconds),  was  used  to  search  and 
count  the  total  number  of  vessels  and  subsequently  finish  the  investigation  reports.  The 
total  number  of  errors  in  counting  vessels  was  nearly  15  per  cent  less  under  fixed 
compared  to  manual  sensor  control.  Hence,  automation  improved  accuracy  on  this  task 
compared  to  no  automation.  However,  this  may  be  the  consequence  of  additional  time 
spent  on  this  task.  The  proportion  of  time  spent  searching  and  counting  the  total  number 
of  vessels  in  areas  was  significantly  longer  under  fixed  than  manual  sensor  control,  with 
participants  using  130  seconds  for  this  part  of  the  reporting  task  in  the  former  condition, 
nearly  double  that  in  the  latter. 

The  intention  of  the  fixed  time  to  complete  both  tasks  was  to  implement  a  standardised 
approach  for  all  sensor  conditions  and  reflect  a  real  world  time  pressure  as  suggested  by 
our  SMEs.  However,  with  most  participants  taking  close  to  the  full  time  to  complete  both 
tasks,  the  fixed  time  acted  as  a  confounding  variable  with  regard  to  total  vessel  count 
times  and  subsequent  total  reporting  times.  In  fixed  sensor  control,  the  sensor  was 
designed  to  hold  at  a  set  pitch  and  azimuth  that  would  allow  all  vessels  in  the  area  around 
the  VOI  to  be  captured  in  its  FOV  as  the  UAV  flew  on  its  designated  path.  This  occurred 
over  a  time  period  of  145  seconds,  although  the  same  vessels  would  have  begun  to 
reappear  in  the  FOV  of  the  sensor  just  before  the  UAV  itself  completed  one  full  flight  lap. 
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As  a  result,  participants  in  this  condition  had  to  use  nearly  all  of  the  available  time  in  order 
to  see  all  of  the  vessels  in  the  area.  On  the  other  hand,  under  manual  sensor  control  the 
participants  only  had  the  time  available  after  completing  the  VOI  task,  nearly  all  of  which 
was  utilised.  Had  we  increased  the  time  to  do  both  tasks  across  conditions  and  given  the 
opportunity  for  the  fixed  sensor  control  to  capture  all  of  the  area  in  its  FOV  a  second  or 
third  time,  we  may  have  seen  different  outcomes  when  comparing  fixed  to  manual  sensor 
control. 

In  this  study,  the  fixed  sensor  condition  was  set  up  such  that  if  participants  noted  the  first 
vessel  seen  in  the  sensor's  FOV,  they  could  systematically  count  the  other  vessels  as  they 
appeared  until  they  observed  the  first  vessel  again.  The  advantage  of  this  set  up  was  lost 
when  operating  under  no  automation.  Under  manual  sensor  control  participants  were 
forced  to  utilise  their  own  search  methods  which,  in  addition  to  less  time  left  to  search 
after  reporting  on  the  VOI,  was  found  to  result  in  increased  counting  error. 

Aligned  with  findings  from  Dixon  et  al.  (2005),  applying  a  fixed  LOA  to  the  sensor  control 
also  improved  performance  on  the  system  status  task  compared  to  having  no  automation. 
In  this  context,  reducing  the  task  demand  of  sensor  manipulation  presumably  allowed 
improved  response  times  to  system  status  changes,  although  the  effect  on  miss  rates  was 
negligible. 


4.2  Adaptable  automation 

The  current  study  found  that  performance  on  reporting  tasks  improved  with  the  ability  to 
change  between  manual  and  fixed  modes  in  adaptable  sensor  control  compared  to  only 
using  manual  sensor  control.  Similar  trends  comparing  adaptable  automation  to  no 
automation  have  been  found  for  other  UAV  tasks  (Miller  et  al.,  2011;  Shaw  et  al.,  2010) 

When  under  adaptable  control,  the  sensor  operation  started  in  fixed  mode,  which  enabled 
participants  to  immediately  report  on  the  VOI  without  having  to  conduct  a  search  for  it. 
Consequently,  reduced  reporting  times  and  error  were  found  in  the  VOI  reporting  task 
under  adaptable  compared  to  manual  sensor  control,  which  supported  the  hypothesis 
proposed. 

The  total  vessels  search  time  under  adaptable  sensor  control  was  significantly  longer  than 
that  under  manual  control,  which  may  have  been  a  result  of  less  time  spent  initially 
locating  and  reporting  on  the  VOI.  Providing  additional  support  for  hypothesis  two, 
adaptable  sensor  control  did,  however,  produce  faster  total  reporting  time  compared  to 
manual  control.  Interestingly,  total  reporting  time  was  also  faster  for  adaptable  compared 
to  fixed  control,  which  indicates  that  participants  must  have  used  the  mode  change  at 
various  times  in  the  adaptable  control  condition,  otherwise  total  reporting  times  for  that 
condition  would  have  been  similar  to  those  for  the  fixed  control  condition  (see  4.3  and  4.4). 

Participants  had  the  option  of  using  the  sensor  automation  or  not  under  adaptable  sensor 
control  and  executed  mode  changes  when  they  felt  it  to  be  necessary.  However,  it  was 
found  that  the  option  to  change  modes  may  have  affected  response  times  to  system  status 
changes  such  that  they  were  slower  than  under  fixed  sensor  control  and  in  fact  very 
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similar  to  response  times  under  manual  control.  Here  lies  one  of  the  criticisms  of 
adaptable  automation.  While  a  LOA  is  designed  to  reduce  the  workload  of  the  human,  the 
human  having  to  designate  a  LOA  to  the  system  can  intermittently  increase  workload  and 
affect  performance  on  tasks  (Miller  &  Parasuraman,  2007) 


4.3  Benefits  of  adaptable  automation 

Although  Kidwell  et  al.  (2012)  found  improved  accuracy  but  with  marginally  increased 
response  time  in  a  change  detection  task  for  adaptable  compared  to  adaptive  automation, 
the  proposition  that  we  would  find  the  same  trend  comparing  adaptable  to  fixed 
automation  across  different  tasks  was  not  completely  supported.  Although  VOI  reporting 
time  was  found  to  be  longer  under  adaptable  compared  to  fixed  sensor  control  (see  4.4), 
the  total  reporting  time  was  actually  faster  under  adaptable  control,  yet  differences  in 
error  in  both  reporting  tasks  were  negligible  between  the  two  conditions.  In  opposition  to 
the  effect  observed  with  regard  to  system  status  responses,  changing  modes  under 
adaptable  sensor  control  did  not  have  a  detrimental  effect  on  total  reporting  time  in  this 
study.  This  was  contrary  to  the  thought  that  user-delegated  automation  might  affect 
performance  on  tasks  (Miller  &  Parasuraman,  2007).  Given  total  reporting  time  comprised 
the  times  required  to  perform  two  separate  reporting  tasks,  interest  lies  in  how 
participants  utilised  adaptable  automation  to  best  perform  these  tasks. 


4.4  Adaptable  automation  use 

It  was  anticipated  that  under  adaptable  control  in  high  density  areas  fixed  mode  would  be 
used  more  than  manual  mode,  but  this  was  not  the  case.  We  found  no  difference  in  mode 
use  as  a  consequence  of  area  density,  and  additionally  discovered  that  participants  spent 
nearly  30  seconds  more  in  manual  compared  to  fixed  mode  overall.  This  was  a  unique 
finding,  and  contradicted  common  thought  that  over-reliance  and  complacency  might 
result  in  automation  being  exploited  (Lee,  2008). 

It  was  revealed  that  under  adaptable  control  for  the  VOI  reporting  task,  participants  spent 
more  time  in  fixed  mode  compared  to  manual  mode.  As  adaptable  control  started  in  fixed 
mode,  it  was  likely  that  participants  would  use  the  initially  allotted  35  seconds  with  the 
sensor  focused  on  the  VOI,  and  this  was  evident  in  the  findings.  Only  an  average  of  three 
more  seconds  were  spent  in  fixed  mode  beyond  this  35  second  time  period,  which  may 
have  been  reflective  of  participants  still  inputting  details  on  their  investigation  report 
before  engaging  in  a  mode  change. 

Fixed  mode  was  not  the  only  mode  used  for  the  VOI  reporting  task.  Under  adaptable 
control,  participants  spent  an  average  of  17  seconds  in  manual  mode  when  attending  to 
this  task.  Given  that  under  fixed  control,  VOI  reporting  error  was  no  different  than  that 
under  adaptable  control,  it  is  not  entirely  clear  why  participants  might  have  used  extra 
modes  here.  Perhaps  as  a  result  of  being  instructed  that  the  VOI  task  was  the  highest 
priority,  participants  felt  a  need  to  utilise  mode  changing  to  ensure  reduced  error  in  VOI 
reporting.  For  this  task,  adaptable  control  may  be  considered  less  efficient  than  fixed 
sensor  control.  Speed  of  VOI  reporting  might  have  been  traded  for  possible  accuracy,  but 
without  any  actual  improvements  in  the  latter  (Miller  &  Parasuraman,  2007). 
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For  the  total  number  of  vessels  reporting  task,  more  time  was  actually  spent  in  manual 
compared  to  fixed  mode  under  adaptable  sensor  control,  with  reduced  reporting  time 
overall  but  no  increase  in  error  compared  to  under  fixed  control.  Here  the  ability  to  change 
the  LOA  of  the  sensor  according  to  changing  task  requirement  led  to  superior  performance 
compared  to  that  associated  with  fixed  sensor  control,  a  similar  trend  to  those  found  by 
others  (Miller  et  al.,  2011;  Parasuraman  et  al.,  2009;  Shaw  et  al.,  2010). 


4.5  Density 

Previous  research  used  a  differing  number  of  surface  vehicles  to  count  between  areas  in 
order  to  differentiate  low  and  high  workload  (Miller  et  al.,  2011;  Shaw  et  al.,  2010),  and  this 
design  was  followed  in  the  current  study.  Similar  trends  to  Miller  et  al.  (2011)  and  Shaw  et 
al.  (2010)  were  also  found  in  the  current  study  whereby  low  density  produced  faster  VOI 
and  total  reporting  times  and  additionally  was  perceived  as  less  workload  compared  to 
high  density.  However,  there  were  also  some  anomalies.  There  was  no  difference  between 
area  density  for  VOI  reporting  error  overall,  which  may  have  been  be  reflective  of  fixed 
and  adaptable  control  already  having  the  sensor  located  at  the  VOI  initially  and  therefore 
being  unaffected  by  the  number  of  other  vessels.  On  the  other  hand,  in  manual  control 
area  density  was  more  likely  to  be  a  confounding  factor  as  participants  had  to  manipulate 
the  sensor  to  the  VOI  amongst  differing  numbers  of  other  vessels,  yet  it  did  not  have  any 
effect.  In  addition,  error  in  the  total  number  of  vessels  counted  in  fixed  sensor  control  was 
actually  higher  in  low  compared  to  high  area  density.  This  suggests  that  changing  the 
number  of  vessels  in  an  area  will  not  necessarily  correspond  to  changes  in  overall 
workload.  A  vigilance  requirement  (Cummings,  Mastracchio,  Thornburg,  &  Mkrtchyan, 
2013)  may  have  inadvertently  been  introduced  in  fixed  sensor  control,  where  the  constant 
monitoring  of  sparse  area  without  being  able  to  manipulate  the  sensor  may  have  increased 
the  error  in  the  total  number  of  vessels  counted.  Conversely,  in  high  area  density 
participants  were  constantly  seeing  new  vessels  and  therefore  receiving  constant 
stimulation  to  count  vessel  numbers.  A  similar  effect  was  found  with  regard  to  system 
status  response  times  overall,  with  faster  response  time  in  high  compared  to  low  density. 
It  may  have  been  the  case  that  visually  scanning  sparse  areas  while  also  monitoring 
gauges  for  changes  could  actually  be  more  difficult  than  continually  coming  across  new 
vessels  and  counting  them  while  also  noticing  any  changes  in  gauges. 


4.6  Condition  preferences 

Participants  preferred  adaptable  control  over  other  sensor  control  conditions,  which  was 
consistent  with  their  pattern  of  performance  in  reporting  tasks  across  control  conditions. 
Although  fixed  sensor  control  produced  poorer  performance  with  regard  to  total  reporting 
time  compared  to  adaptable  sensor  control,  it  did  improve  system  status  response  time 
and  was  perceived  as  creating  less  workload  than  adaptable  or  manual  control.  Despite 
this,  fixed  control  was  the  least  preferred  sensor  control  condition.  This  may  have  been 
due  to  participants  not  trusting  the  automation  and  therefore  not  believing  it  was  most 
suitable  for  the  task  at  hand  (Lee,  2006). 


36 


UNCLASSIFIED 


UNCLASSIFIED 


DST-Group-TR-2953 


4.7  Limitations 

4.7.1  Experimental  design 

A  potential  limitation  of  this  study  was  the  time  allowed  to  complete  the  reporting  tasks. 
In  the  fixed  sensor  control  condition,  a  35  second  time  on  the  VOI  and  subsequent  scan  of 
the  area  were  implemented  from  consultation  with  SMEs.  However,  a  ceiling  effect  was 
inadvertently  created  whereby  the  majority  of  participants  submitted  the  investigation 
report  just  before  the  finish  time  of  an  area,  which  was  after  one  complete  scan.  If  the 
overall  time  in  areas  had  been  increased  and  several  scans  of  an  area  could  have  taken 
place,  there  may  have  been  different  performance  effects,  particularly  with  regard  to  the 
fixed  and  manual  conditions  for  which  differences  in  total  reporting  times  were  negligible 
in  the  current  study. 

Under  adaptable  sensor  control,  participants  were  able  to  switch  freely  between  fixed  and 
manual  modes  through  a  single  mode  button  activation.  However,  when  participants 
switched  back  to  fixed  from  manual  mode  the  sensor  only  moved  back  to  the  pre¬ 
programmed  set  angle,  regardless  of  whether  they  were  investigating  the  VOI  or  finding 
the  total  number  of  vessels  in  the  area.  Consequently,  if  participants  accidentally  pressed 
the  mode  button  while  the  sensor  was  located  on  the  VOI  initially,  they  would  need  to 
manually  manipulate  the  sensor  if  they  wished  to  continue  viewing  the  VOI,  since 
pressing  the  mode  button  a  second  time  would  not  take  the  sensor  back  to  that  location.  If 
participants  had  the  ability  to  select  whether  the  sensor  returned  to  the  VOI  or  adopted  the 
set  scanning  angle,  this  would  have  provided  them  with  further  control  over  the  use  of  the 
automation,  potentially  improving  performance  and  removing  noise  resulting  from 
accidental  mode  button  presses. 

Parasuraman  et  al.  (2000)  suggested  that  LOAs  might  be  applied  along  a  continuum  as 
opposed  to  either  no  automation  or  full  automation,  which  was  applied  in  the  current 
study.  The  number  of  sensor  conditions  was  chosen  in  the  current  study  to  avoid 
considerable  complexity  and  the  need  to  collect  considerably  more  data  to  achieve  the 
power  required  for  additional  analyses.  An  additional  intermediate  LOA  or  MBC 
condition,  whereby  the  system  asked  participants  to  accept  a  change  in  automated  surface 
searching  before  continuing,  may  have  provided  further  insight  into  how  adaptable 
automation  might  be  used. 

4.7.2  Simulation  software 

The  VBS2  software  provided  the  simulated  environment;  however,  there  were  substantial 
limitations  in  modelling  the  necessary  experiment  functionality.  Consequently,  significant 
scripting  was  required  involving  the  development  of  both  internal  VBS2  scripts  and 
additional  external  scripts  written  in  Groovy.  Internal  VBS2  development  was  limited  to 
the  scripting  interface  and  did  not  involve  the  development  of  plugins,  primarily  due  to 
the  limited  timeframe  available  for  development  and  the  lack  of  VBS2  development 
experience.  Additionally,  many  initial  design  decisions  required  adjustment  due  to 
incorrect  or  misleading  VBS2  documentation.  Further  time  and  provision  of  standardised 
software  documentation  may  have  mitigated  some  of  these  restrictions. 
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Existing  UAV  models  provided  by  VBS2  were  not  adequate  to  meet  the  experiment 
requirements,  as  it  was  not  possible  to  modify  the  user  control  over  the  sensor  and 
difficult  to  adapt  the  provided  HUD.  To  overcome  this,  an  existing  VBS2  UAV  model  was 
used  in  combination  with  a  custom  sensor  view.  A  further  complication  arose  from  the 
custom  sensor  view  responding  to  engine  heat  haze  from  the  VBS2  Scan  Eagle  model, 
which  restricted  visibility  and  would  have  created  a  confounding  variable  experimentally. 
This  was  controlled  through  the  creation  of  the  sensor  view  well  below  the  actual  location 
of  the  UAV,  but  at  the  same  time  may  have  also  inadvertently  reduced  the  generalisability 
of  the  UAV  system  to  other  simulated  Scan  Eagle  models. 

Performance  issues  arose  when  modelling  a  large  number  of  vessels  via  multiple  areas 
located  within  the  single  VBS2  scenario.  Programmatically  creating  and  removing  vessels 
between  areas  was  employed  to  ensure  participants  only  sighted  vessels  within  a 
particular  area,  but  this  implementation  also  became  necessary  to  eliminate  an 
unacceptably  slow  frame  rate  and  a  resulting  slowed  sensor  responsiveness  when  all 
vessels  were  present. 

For  the  reporting  task,  investigation  reports  were  initially  implemented  within  VBS2  using 
a  dialog  control.  This  used  VBS2  custom  events  to  coordinate  the  display  and  submission 
of  reports  with  the  scenario  tasking.  As  the  modal  nature  of  the  dialog  did  not  allow 
concurrent  user  interaction  with  reporting  and  sensor  control,  a  second  instance  of  VBS2  in 
multiplayer  mode  was  required  to  run  on  a  second  computer  to  maintain  user  control  of 
the  sensor  while  simultaneously  providing  input  to  the  report.  Upon  discovery  that  VBS2 
did  not  propagate  custom  events  over  a  multiplayer  connection,  the  reports  required  re¬ 
implementing  external  to  VBS2,  utilising  the  custom  Groovy  scripts.  It  could  have  been 
more  beneficial  to  implement  custom  Groovy  scripts  from  the  outset  of  the  design  had 
there  been  prior  knowledge  of  some  of  the  restrictions  in  creating  custom  events  in  VBS2. 


4.8  Future  research 

The  current  study  completely  automated  all  flight  and  navigation  control,  and  only 
manipulated  sensor  control  when  the  UAV  was  in  flight  in  the  specified  area  to  be 
investigated.  Future  research  might  consider  having  an  additional  change  in  the 
environment,  such  as  the  VOI  moving  outside  of  the  specified  area,  which  requires  the 
user  to  set  new  flight  paths  depicted  by  waypoints  during  an  experimental  session.  In  this 
way,  a  more  real  world  setting  would  be  introduced  in  the  simulation,  and  it  would  be 
useful  to  then  assess  how  adding  a  task  from  the  navigation  control  loop  might  impact 
performance  in  higher  level  tasks  of  sensor  manipulation  and  reporting.  Additionally, 
given  that  in  the  current  study  performance  in  the  system  status  task  deteriorated  when 
sensor  control  was  not  completely  automated,  it  would  be  practical  to  examine  whether 
further  impacts  are  resultant  from  additional  navigational  requirements. 

Although  the  majority  of  previous  research  has  focused  on  tasks  with  multiple  UAV 
control,  tasks  have  been  varied  and  not  directly  comparable.  Future  research  following  this 
study  might  consider  simply  building  on  the  number  of  required  tasks  with  single  UAV 
control.  It  is  proposed  that  this  is  necessary  to  gain  clearer  insight  into  use  of  automation 
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at  the  sensor  control  level,  before  potentially  adding  the  complexity  of  more  than  one  UAV 
to  be  managed  by  an  operator. 

To  support  the  operator  in  their  situational  awareness,  the  sensor  HUD  included  a  mini 
map  as  depicted  in  Figure  2.  The  map  included  the  location  of  the  VOI,  the  flight  path, 
location  and  direction  of  the  UAV,  and  a  direction  indicator  of  the  sensor  to  assist  the 
operator  with  understanding  where  the  sensor  was  directed.  During  development,  the  use 
of  a  dynamic  length  sensor  indicator  was  explored,  where  the  location  of  the  indicator 
arrow  head  corresponded  to  centre  point  of  the  sensor  view,  essentially  indicating  the 
exact  location  the  operator  was  viewing.  Early  trials  of  this  feature,  in  consultation  with 
SMEs,  presented  concerns  that  this  would  make  the  search  task  too  easy  for  the  operators 
and  was  subsequently  removed. 

An  investigation  into  the  established  and  researched  techniques  for  providing  the  operator 
with  an  indication  of  the  sensor  view  location  has  not  been  undertaken.  However,  the 
approach  described  above  has  not  been  seen  before  by  the  experiment  team  or  the  SMEs. 
Given  that  the  dynamic  length  sensor  indicator  appeared  to  improve  the  ability  of  the 
operator  to  perform  search  tasks  during  pre-experiment  trials,  and  the  positive  feedback 
from  the  SMEs,  it  may  be  worthwhile  exploring  this  approach  further.  It  may  also  be 
worth  exploring  the  different  ways  of  displaying  a  representation  of  the  complete  FOV  of 
the  sensor  on  the  mini  map  in  addition  to  the  centre  point  indicator. 
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5.  Conclusion 

The  current  study  has  shown  that  automation  use  in  an  UAV  context  supported  operators 
to  better  perform  their  required  tasks.  When  automation  was  applied  to  the  sensor  control, 
it  was  shown  to  improve  overall  performance  in  reporting  and  system  status  tasks 
compared  to  when  no  automation  was  applied.  However,  applying  an  unchangeable  LOA 
to  sensor  control  also  produced  poorer  performance  compared  to  when  operators  had 
adaptable  control  and  could  manipulate  the  LOA  of  the  sensor.  Adaptable  automation  in 
this  context  better  suited  the  need  for  completing  different  types  of  reporting  when 
conducting  surface  search  tasks,  where  overall  reporting  times  were  reduced  without 
increasing  error  compared  to  fixed  automation.  Yet  benefits  were  not  observed  across  all 
tasks;  response  times  and  miss  rates  of  system  status  changes  were  much  the  same 
between  adaptable  and  manual  sensor  control.  It  is  suggested  that  automating  sensor 
control  in  UAVs  will  improve  performance  compared  to  when  no  automation  is  used,  but 
an  adaptable  approach  may  provide  some  additional  performance  benefits  and  greater 
user  satisfaction. 
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